[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:11 dist-test-slave-4tdj krb5kdc[29725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:11 dist-test-slave-4tdj krb5kdc[29725](info): set up 2 sockets
Aug 11 22:20:11 dist-test-slave-4tdj krb5kdc[29725](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 22:20:12.158983 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:12 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:12.286367 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42813
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:42813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:12.425272 29749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:12.425603 29749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:12.425664 29749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:12.430228 29749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:12.430366 29749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:12.430399 29749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:12.430418 29749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:12.430438 29749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:12.436398 29749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:42813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:12.437513 29749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:12.438607 29749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:12.448678 29761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:12.448844 29759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:12.448807 29763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:12.448915 29749 server_base.cc:1034] running on GCE node
I20260811 22:20:12.449674 29749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:12.450898 29749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:12.452065 29749 hybrid_clock.cc:648] HybridClock initialized: now 1786486812452039 us; error 44 us; skew 500 ppm
Aug 11 22:20:12 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:12.455106 29749 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:12.456049 29749 webserver.cc:460] Webserver started at http://127.29.3.126:37961/ using document root <none> and password file <none>
I20260811 22:20:12.456641 29749 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:12.456712 29749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:12.456920 29749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:12.459156 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "cf13eb327579454883a67637edd68101"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "9de91044e7d673eedc75cc8e5e94c142"
server_key_iv: "f214954384a6e859660b9edd5f5fe36e"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.459769 29749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "cf13eb327579454883a67637edd68101"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "9de91044e7d673eedc75cc8e5e94c142"
server_key_iv: "f214954384a6e859660b9edd5f5fe36e"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.463761 29749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:20:12.466379 29776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:12.467656 29749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:20:12.467885 29749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "cf13eb327579454883a67637edd68101"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "9de91044e7d673eedc75cc8e5e94c142"
server_key_iv: "f214954384a6e859660b9edd5f5fe36e"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.468014 29749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:12.482864 29749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:12.487048 29749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:12.487358 29749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:12.498100 29749 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:42813
I20260811 22:20:12.498370 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:42813 every 8 connection(s)
I20260811 22:20:12.499215 29749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:12.502153 29873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:12.504818 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 29749
I20260811 22:20:12.504966 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:12.506265 29709 external_mini_cluster.cc:1428] Setting key b7c33a6ecdfc59c4f65fe6a474beeb68
I20260811 22:20:12.508419 29873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101: Bootstrap starting.
I20260811 22:20:12.511488 29873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:12.512238 29873 log.cc:824] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:12.514232 29873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101: No bootstrap required, opened a new log
I20260811 22:20:12.518023 29873 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf13eb327579454883a67637edd68101" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42813 } }
I20260811 22:20:12.518249 29873 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:12.518347 29873 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf13eb327579454883a67637edd68101, State: Initialized, Role: FOLLOWER
I20260811 22:20:12.518818 29873 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf13eb327579454883a67637edd68101" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42813 } }
I20260811 22:20:12.519032 29873 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:12.519105 29873 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:12.519209 29873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:12.520115 29873 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf13eb327579454883a67637edd68101" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42813 } }
I20260811 22:20:12.520609 29873 leader_election.cc:302] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf13eb327579454883a67637edd68101; no voters:
I20260811 22:20:12.521044 29873 leader_election.cc:288] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:12.521155 29883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:12.521387 29883 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [term 1 LEADER]: Becoming Leader. State: Replica: cf13eb327579454883a67637edd68101, State: Running, Role: LEADER
I20260811 22:20:12.521733 29883 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf13eb327579454883a67637edd68101" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42813 } }
I20260811 22:20:12.522476 29873 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:12.523375 29885 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf13eb327579454883a67637edd68101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf13eb327579454883a67637edd68101" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42813 } } }
I20260811 22:20:12.523507 29885 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:12.523847 29888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf13eb327579454883a67637edd68101. Latest consensus state: current_term: 1 leader_uuid: "cf13eb327579454883a67637edd68101" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf13eb327579454883a67637edd68101" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42813 } } }
I20260811 22:20:12.523943 29888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:12.524237 29896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 11 22:20:12 dist-test-slave-4tdj krb5kdc[29725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:12.528017 29896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:12.531770 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:12.509586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.29.3.126:42813)
0811 22:20:12.510026 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:12.510043 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:12.511262 (+ 1219us) server_negotiation.cc:407] Connection header received
0811 22:20:12.514510 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:12.514531 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:12.514848 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:12.515167 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:12.517305 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:12.518259 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:12.519700 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:12.520171 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:12.520440 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:12.525609 (+ 5169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:12.525616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:12.525633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:12.525668 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:12.527734 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:12.528453 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:12.528459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:12.528465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:12.528616 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:12.529051 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:12.529055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:12.529057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:12.529697 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:12.529888 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:12.530565 (+ 677us) server_negotiation.cc:299] Negotiation successful
0811 22:20:12.530872 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":126,"threads_started":1}
I20260811 22:20:12.534693 29896 catalog_manager.cc:1347] Generated new cluster ID: 3bf90c90d4bd4812aebfc248444ddc24
I20260811 22:20:12.534773 29896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:12.547294 29896 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:12.548594 29896 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:12.555666 29896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf13eb327579454883a67637edd68101: Generated new TSK 0
I20260811 22:20:12.556540 29896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:12.593629 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42813
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:12.717321 29929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:12.717597 29929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:12.717657 29929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:12.721676 29929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:12.721745 29929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:12.721827 29929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:12.727257 29929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:12.728322 29929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:12.729342 29929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:12.739001 29936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:12.739001 29941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:12.739221 29937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:12.739378 29929 server_base.cc:1034] running on GCE node
I20260811 22:20:12.739821 29929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:12.740517 29929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:12.741684 29929 hybrid_clock.cc:648] HybridClock initialized: now 1786486812741665 us; error 29 us; skew 500 ppm
Aug 11 22:20:12 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:12.744469 29929 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:12.745388 29929 webserver.cc:460] Webserver started at http://127.29.3.65:35745/ using document root <none> and password file <none>
I20260811 22:20:12.745940 29929 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:12.746001 29929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:12.746215 29929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:12.748307 29929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "6d5ea12a101f431c9b8e0923862ab5af"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "1612018b9adbced1b177af856a212fdc"
server_key_iv: "ba841be722616cd319356666322a780e"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.748823 29929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "6d5ea12a101f431c9b8e0923862ab5af"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "1612018b9adbced1b177af856a212fdc"
server_key_iv: "ba841be722616cd319356666322a780e"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.752467 29929 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:20:12.754832 29955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:12.756006 29929 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:12.756105 29929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "6d5ea12a101f431c9b8e0923862ab5af"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "1612018b9adbced1b177af856a212fdc"
server_key_iv: "ba841be722616cd319356666322a780e"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.756208 29929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:12.776535 29929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:12.779384 29929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:12.779538 29929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:12.780095 29929 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:12.780922 29929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:12.780972 29929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:12.781010 29929 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:12.781030 29929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:12.790755 29929 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:42075
I20260811 22:20:12.790796 30162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:42075 every 8 connection(s)
I20260811 22:20:12.791651 29929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
Aug 11 22:20:12 dist-test-slave-4tdj krb5kdc[29725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:12.801017 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 29929
I20260811 22:20:12.801111 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:12.801398 29709 external_mini_cluster.cc:1428] Setting key 3c382ba1b0f1e4fb9b5d85af400b05f6
I20260811 22:20:12.804764 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:12.793426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38489 (local address 127.29.3.126:42813)
0811 22:20:12.793544 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:12.793548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:12.794610 (+ 1062us) server_negotiation.cc:407] Connection header received
0811 22:20:12.795502 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:12.795505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:12.795568 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:12.795641 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:12.796818 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:12.797355 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:12.798612 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:12.798859 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:12.798955 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:12.801301 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:12.801305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:12.801308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:12.801339 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:12.802785 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:12.803295 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:12.803299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:12.803300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:12.803401 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:12.803703 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:12.803706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:12.803708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:12.803927 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:12.804021 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:12.804483 (+ 462us) server_negotiation.cc:299] Negotiation successful
0811 22:20:12.804608 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 22:20:12.805543 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:12.793766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42813 (local address 127.29.3.65:38489)
0811 22:20:12.794466 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:12.794510 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:12.795296 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:12.795833 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:12.795839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:12.796183 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:12.796666 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:12.796680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:12.797478 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:12.797481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:12.798490 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:12.798501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:12.799005 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:12.799009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:12.799172 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:12.799691 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:12.799708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:12.801128 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:12.802894 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:12.802900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:12.802914 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:12.803191 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:12.803499 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:12.803501 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:12.803503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:12.803627 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:12.804011 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:12.804016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:12.804262 (+ 246us) client_negotiation.cc:769] Sending connection context
0811 22:20:12.804417 (+ 155us) client_negotiation.cc:240] Negotiation successful
0811 22:20:12.804670 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":118,"threads_started":1}
I20260811 22:20:12.806804 30163 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42813
I20260811 22:20:12.807080 30163 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:12.807629 30163 heartbeater.cc:511] Master 127.29.3.126:42813 requested a full tablet report, sending...
I20260811 22:20:12.809455 29807 ts_manager.cc:194] Registered new tserver with Master: 6d5ea12a101f431c9b8e0923862ab5af (127.29.3.65:42075)
I20260811 22:20:12.811059 29807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:38489
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:12.830720 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42813
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:12.950038 30175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:12.950304 30175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:12.950369 30175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:12.954250 30175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:12.954324 30175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:12.954403 30175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:12.959601 30175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:12.960704 30175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:12.961689 30175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:12.971761 30181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:12.971815 30182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:12.971921 30175 server_base.cc:1034] running on GCE node
W20260811 22:20:12.971784 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:12.972509 30175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:12.973160 30175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:12.974339 30175 hybrid_clock.cc:648] HybridClock initialized: now 1786486812974320 us; error 31 us; skew 500 ppm
Aug 11 22:20:12 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:12.977026 30175 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:12.977921 30175 webserver.cc:460] Webserver started at http://127.29.3.66:45555/ using document root <none> and password file <none>
I20260811 22:20:12.978503 30175 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:12.978566 30175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:12.978755 30175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:12.980749 30175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "7cc22bdad9b84d50824315c43ab4bfe8"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "0492ebeec70c09359b331a32c2a5a8f7"
server_key_iv: "64943e61f3ef686c37f9d7a13c67519c"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.981248 30175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "7cc22bdad9b84d50824315c43ab4bfe8"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "0492ebeec70c09359b331a32c2a5a8f7"
server_key_iv: "64943e61f3ef686c37f9d7a13c67519c"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.984933 30175 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260811 22:20:12.987370 30197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:12.988520 30175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:12.988617 30175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "7cc22bdad9b84d50824315c43ab4bfe8"
format_stamp: "Formatted at 2026-08-11 22:20:12 on dist-test-slave-4tdj"
server_key: "0492ebeec70c09359b331a32c2a5a8f7"
server_key_iv: "64943e61f3ef686c37f9d7a13c67519c"
server_key_version: "encryptionkey@0"
I20260811 22:20:12.988699 30175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:13.006767 30175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:13.010265 30175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:13.010514 30175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:13.011103 30175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:13.011931 30175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:13.011978 30175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:13.012018 30175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:13.012039 30175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:13.024031 30175 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:43805
I20260811 22:20:13.024523 30419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:43805 every 8 connection(s)
I20260811 22:20:13.025208 30175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:13.027171 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30175
I20260811 22:20:13.027285 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:13.027586 29709 external_mini_cluster.cc:1428] Setting key 2eb8c1c4ed26231fb1193018e88f82dd
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486812, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:20:13.040935 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:13.029034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:57617 (local address 127.29.3.126:42813)
0811 22:20:13.029155 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:13.029160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:13.030352 (+ 1192us) server_negotiation.cc:407] Connection header received
0811 22:20:13.030400 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:13.030402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:13.030446 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:13.030506 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:13.032318 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.032842 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.033993 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.034224 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.034339 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:13.037181 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:13.037184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:13.037186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:13.037213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:13.038746 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.039265 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.039268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.039271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.039352 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.039708 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.039712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.039714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.039910 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:13.039999 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:13.040537 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 22:20:13.040724 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
I20260811 22:20:13.041807 30424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:13.027945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42813 (local address 127.29.3.66:57617)
0811 22:20:13.029067 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:13.029183 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:13.030060 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:13.030743 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:13.030754 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:13.031441 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:13.032139 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:13.032158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.032962 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.032965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:13.033869 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:13.033879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.034389 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.034392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:13.034579 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:13.035158 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:13.035174 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:13.036995 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:13.038870 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:13.038876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:13.038895 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:13.039159 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:13.039441 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:13.039443 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:13.039445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:13.039602 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:13.040012 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:13.040020 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:13.040292 (+ 272us) client_negotiation.cc:769] Sending connection context
0811 22:20:13.040471 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 22:20:13.040846 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":153,"threads_started":1}
I20260811 22:20:13.043071 30422 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42813
I20260811 22:20:13.043350 30422 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:13.043962 30422 heartbeater.cc:511] Master 127.29.3.126:42813 requested a full tablet report, sending...
I20260811 22:20:13.045109 29807 ts_manager.cc:194] Registered new tserver with Master: 7cc22bdad9b84d50824315c43ab4bfe8 (127.29.3.66:43805)
I20260811 22:20:13.045696 29807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:57617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:13.057753 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42813
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:13.181530 30427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:13.181813 30427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:13.181874 30427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:13.186000 30427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:13.186071 30427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:13.186157 30427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:13.191988 30427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42813
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:13.193202 30427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:13.194258 30427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:13.204310 30438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:13.204356 30436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:13.204310 30434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:13.204712 30427 server_base.cc:1034] running on GCE node
I20260811 22:20:13.205092 30427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:13.205750 30427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:13.206934 30427 hybrid_clock.cc:648] HybridClock initialized: now 1786486813206901 us; error 44 us; skew 500 ppm
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486813, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:13.210350 30427 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:13.211261 30427 webserver.cc:460] Webserver started at http://127.29.3.67:42649/ using document root <none> and password file <none>
I20260811 22:20:13.211803 30427 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:13.211861 30427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:13.212038 30427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:13.214160 30427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "1295d348a6514fa2bb8d4e7f2eee5769"
format_stamp: "Formatted at 2026-08-11 22:20:13 on dist-test-slave-4tdj"
server_key: "950a9137c85fe0dab24898ef05952ded"
server_key_iv: "d2bebb3e147570694345f76cd01e4261"
server_key_version: "encryptionkey@0"
I20260811 22:20:13.214754 30427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "1295d348a6514fa2bb8d4e7f2eee5769"
format_stamp: "Formatted at 2026-08-11 22:20:13 on dist-test-slave-4tdj"
server_key: "950a9137c85fe0dab24898ef05952ded"
server_key_iv: "d2bebb3e147570694345f76cd01e4261"
server_key_version: "encryptionkey@0"
I20260811 22:20:13.218576 30427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:20:13.221095 30458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:13.222433 30427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:13.222549 30427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "1295d348a6514fa2bb8d4e7f2eee5769"
format_stamp: "Formatted at 2026-08-11 22:20:13 on dist-test-slave-4tdj"
server_key: "950a9137c85fe0dab24898ef05952ded"
server_key_iv: "d2bebb3e147570694345f76cd01e4261"
server_key_version: "encryptionkey@0"
I20260811 22:20:13.222656 30427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:13.234592 30427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:13.237804 30427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:13.237988 30427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:13.238636 30427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:13.239499 30427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:13.239549 30427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:13.239603 30427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:13.239632 30427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:13.250108 30427 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:45939
I20260811 22:20:13.250131 30670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:45939 every 8 connection(s)
I20260811 22:20:13.251088 30427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:13.254904 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30427
I20260811 22:20:13.255012 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:13.255273 29709 external_mini_cluster.cc:1428] Setting key bf20bb1de275caf09862b2c52fbf07c7
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486813, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:13.265599 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:13.253080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:42699 (local address 127.29.3.126:42813)
0811 22:20:13.253232 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:13.253237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:13.254431 (+ 1194us) server_negotiation.cc:407] Connection header received
0811 22:20:13.255618 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:13.255622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:13.255694 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:13.255783 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:13.257380 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.258021 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.259343 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.259620 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.259726 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:13.262445 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:13.262448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:13.262450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:13.262480 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:13.263817 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.264267 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.264269 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.264270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.264339 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.264612 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.264614 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.264615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.264792 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:13.264864 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:13.265363 (+ 499us) server_negotiation.cc:299] Negotiation successful
0811 22:20:13.265470 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:20:13.266587 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:13.253516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42813 (local address 127.29.3.67:42699)
0811 22:20:13.254222 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:13.254276 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:13.255390 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:13.256002 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:13.256010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:13.256584 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:13.257239 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:13.257256 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.258145 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.258148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:13.259200 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:13.259211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.259775 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.259778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:13.259934 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:13.260527 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:13.260544 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:13.262240 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:13.263912 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:13.263917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:13.263930 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:13.264185 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:13.264400 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:13.264402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:13.264403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:13.264544 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:13.264863 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:13.264868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:13.265120 (+ 252us) client_negotiation.cc:769] Sending connection context
0811 22:20:13.265289 (+ 169us) client_negotiation.cc:240] Negotiation successful
0811 22:20:13.265561 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":125,"threads_started":1}
I20260811 22:20:13.267787 30671 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42813
I20260811 22:20:13.268064 30671 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:13.268621 30671 heartbeater.cc:511] Master 127.29.3.126:42813 requested a full tablet report, sending...
I20260811 22:20:13.269855 29809 ts_manager.cc:194] Registered new tserver with Master: 1295d348a6514fa2bb8d4e7f2eee5769 (127.29.3.67:45939)
I20260811 22:20:13.270648 29809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:42699
I20260811 22:20:13.281801 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486813, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:13.309407 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:13.300473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.29.3.65:42075)
0811 22:20:13.300815 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:13.300831 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:13.300857 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:20:13.300913 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:13.300918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:13.301107 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:13.301237 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:13.301871 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.302696 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.303518 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.303898 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.303981 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:13.306378 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:13.306390 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:13.306405 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:13.306432 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:13.307849 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.308238 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.308243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.308244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.308320 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.308520 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.308522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.308523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.308779 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:13.308938 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:13.309086 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 22:20:13.309223 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":114,"threads_started":1}
W20260811 22:20:13.310492 30005 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33948
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[29725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:13.338465 30678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:13.330381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.29.3.65:42075)
0811 22:20:13.330553 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:13.330558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:13.330650 (+ 92us) server_negotiation.cc:407] Connection header received
0811 22:20:13.330785 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:13.330787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:13.330837 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:13.330908 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:13.331582 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.332089 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.332956 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:13.333203 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:13.333286 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:13.335681 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:13.335684 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:13.335686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:13.335721 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:13.337019 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.337441 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.337444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.337445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.337511 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:13.337727 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:13.337729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:13.337730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:13.337887 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:13.338114 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:13.338172 (+ 58us) server_negotiation.cc:299] Negotiation successful
0811 22:20:13.338272 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:20:13.339646 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 29929
I20260811 22:20:13.356369 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30175
I20260811 22:20:13.367836 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30427
I20260811 22:20:13.407752 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 29749
2026-08-11T22:20:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2484 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[30695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[30695](info): set up 2 sockets
Aug 11 22:20:13 dist-test-slave-4tdj krb5kdc[30695](info): commencing operation
krb5kdc: starting...
W20260811 22:20:14.603242 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:14 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:14.726890 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40375
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:40375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:14.839926 30723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:14.840190 30723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:14.840247 30723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:14.844316 30723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:14.844384 30723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:14.844410 30723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:14.844431 30723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:14.844461 30723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:14.849831 30723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:40375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:14.850993 30723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:14.852013 30723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:14.860812 30731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:14.860875 30723 server_base.cc:1034] running on GCE node
W20260811 22:20:14.860844 30739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:14.860812 30735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:14.861661 30723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:14.862855 30723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:14.864038 30723 hybrid_clock.cc:648] HybridClock initialized: now 1786486814864009 us; error 42 us; skew 500 ppm
Aug 11 22:20:14 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486814, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:14.866822 30723 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:14.867733 30723 webserver.cc:460] Webserver started at http://127.29.3.126:42791/ using document root <none> and password file <none>
I20260811 22:20:14.868265 30723 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:14.868323 30723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:14.868497 30723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:14.870651 30723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "1788ea6baec84e008fe591fcf8ce44a1"
format_stamp: "Formatted at 2026-08-11 22:20:14 on dist-test-slave-4tdj"
server_key: "20d4c3345d1d18a7c05e5e3d89eef415"
server_key_iv: "dab6976c240f534f243f7e3c731992d9"
server_key_version: "encryptionkey@0"
I20260811 22:20:14.871181 30723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "1788ea6baec84e008fe591fcf8ce44a1"
format_stamp: "Formatted at 2026-08-11 22:20:14 on dist-test-slave-4tdj"
server_key: "20d4c3345d1d18a7c05e5e3d89eef415"
server_key_iv: "dab6976c240f534f243f7e3c731992d9"
server_key_version: "encryptionkey@0"
I20260811 22:20:14.875032 30723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:20:14.877408 30753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:14.878681 30723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:14.878789 30723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "1788ea6baec84e008fe591fcf8ce44a1"
format_stamp: "Formatted at 2026-08-11 22:20:14 on dist-test-slave-4tdj"
server_key: "20d4c3345d1d18a7c05e5e3d89eef415"
server_key_iv: "dab6976c240f534f243f7e3c731992d9"
server_key_version: "encryptionkey@0"
I20260811 22:20:14.878882 30723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:14.889103 30723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:14.892102 30723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:14.892269 30723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:14.899705 30723 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:40375
I20260811 22:20:14.899737 30865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:40375 every 8 connection(s)
I20260811 22:20:14.900688 30723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:14.903550 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30723
I20260811 22:20:14.903656 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:14.903654 30867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:14.903901 29709 external_mini_cluster.cc:1428] Setting key 0afee91e7737328dea747417a3c4de3f
I20260811 22:20:14.911149 30867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1: Bootstrap starting.
Aug 11 22:20:14 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:14.913682 30867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:14.914597 30867 log.cc:824] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:14.916678 30867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1: No bootstrap required, opened a new log
I20260811 22:20:14.918634 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:14.905386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.29.3.126:40375)
0811 22:20:14.905837 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:14.905855 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:14.905900 (+ 45us) server_negotiation.cc:407] Connection header received
0811 22:20:14.906595 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:14.906621 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:14.907107 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:14.907488 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:14.908441 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:14.909949 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:14.910894 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:14.911566 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:14.911736 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:14.914163 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:14.914168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:14.914183 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:14.914213 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:14.915895 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:14.916354 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:14.916358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:14.916363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:14.916465 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:14.916753 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:14.916756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:14.916757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:14.917154 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:14.917303 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:14.917586 (+ 283us) server_negotiation.cc:299] Negotiation successful
0811 22:20:14.917858 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":123,"threads_started":1}
I20260811 22:20:14.920660 30867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1788ea6baec84e008fe591fcf8ce44a1" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40375 } }
I20260811 22:20:14.920898 30867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:14.920956 30867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1788ea6baec84e008fe591fcf8ce44a1, State: Initialized, Role: FOLLOWER
I20260811 22:20:14.921368 30867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1788ea6baec84e008fe591fcf8ce44a1" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40375 } }
I20260811 22:20:14.921495 30867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:14.921553 30867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:14.921648 30867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:14.922593 30867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1788ea6baec84e008fe591fcf8ce44a1" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40375 } }
I20260811 22:20:14.922932 30867 leader_election.cc:302] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1788ea6baec84e008fe591fcf8ce44a1; no voters:
I20260811 22:20:14.923204 30867 leader_election.cc:288] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:14.923316 30875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:14.923516 30875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [term 1 LEADER]: Becoming Leader. State: Replica: 1788ea6baec84e008fe591fcf8ce44a1, State: Running, Role: LEADER
I20260811 22:20:14.923813 30875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1788ea6baec84e008fe591fcf8ce44a1" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40375 } }
I20260811 22:20:14.924340 30867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:14.924856 30875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1788ea6baec84e008fe591fcf8ce44a1. Latest consensus state: current_term: 1 leader_uuid: "1788ea6baec84e008fe591fcf8ce44a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1788ea6baec84e008fe591fcf8ce44a1" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40375 } } }
I20260811 22:20:14.925089 30875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:14.925325 30878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1788ea6baec84e008fe591fcf8ce44a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1788ea6baec84e008fe591fcf8ce44a1" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40375 } } }
I20260811 22:20:14.925472 30878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:14.925639 30890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:14.929029 30890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:14.934661 30890 catalog_manager.cc:1347] Generated new cluster ID: 09d48ea81ab94eeca145c9ffd7ee1cb9
I20260811 22:20:14.934725 30890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:14.952792 30890 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:14.953761 30890 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:14.964143 30890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1788ea6baec84e008fe591fcf8ce44a1: Generated new TSK 0
I20260811 22:20:14.964902 30890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:14.997570 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40375
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:15.114414 30916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:15.114698 30916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:15.114750 30916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:15.118806 30916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:15.118875 30916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:15.118933 30916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:15.124243 30916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:15.125385 30916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:15.126446 30916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:15.136232 30930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:15.136245 30925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:15.136247 30934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:15.136283 30916 server_base.cc:1034] running on GCE node
I20260811 22:20:15.136857 30916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:15.137454 30916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:15.138607 30916 hybrid_clock.cc:648] HybridClock initialized: now 1786486815138587 us; error 38 us; skew 500 ppm
Aug 11 22:20:15 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:15.141261 30916 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:15.142117 30916 webserver.cc:460] Webserver started at http://127.29.3.65:45609/ using document root <none> and password file <none>
I20260811 22:20:15.142714 30916 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:15.142784 30916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:15.143000 30916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:15.145084 30916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "48f3ec4f772b474387907140d8a5a5a6"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "ed7737602edaafc7853ce009cbbee61a"
server_key_iv: "ec9bbfc31fe770effd2b4142dc898b08"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.145653 30916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "48f3ec4f772b474387907140d8a5a5a6"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "ed7737602edaafc7853ce009cbbee61a"
server_key_iv: "ec9bbfc31fe770effd2b4142dc898b08"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.149317 30916 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:20:15.151706 30945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.152887 30916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:15.152997 30916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "48f3ec4f772b474387907140d8a5a5a6"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "ed7737602edaafc7853ce009cbbee61a"
server_key_iv: "ec9bbfc31fe770effd2b4142dc898b08"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.153088 30916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:15.170231 30916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:15.172995 30916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:15.173177 30916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:15.173768 30916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:15.174640 30916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:15.174700 30916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.174757 30916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:15.174780 30916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.183292 30916 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:39157
I20260811 22:20:15.183324 31169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:39157 every 8 connection(s)
I20260811 22:20:15.184228 30916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:15.186157 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30916
I20260811 22:20:15.186270 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:15.186549 29709 external_mini_cluster.cc:1428] Setting key c75d1d4a04f085edaf16ca23e194cc30
Aug 11 22:20:15 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:20:15.198424 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.186095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:44763 (local address 127.29.3.126:40375)
0811 22:20:15.186272 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.186278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.187208 (+ 930us) server_negotiation.cc:407] Connection header received
0811 22:20:15.188635 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.188638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.188694 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.188780 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:15.189971 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.190561 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.191701 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.191941 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.192033 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.194820 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:15.194823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:15.194825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:15.194852 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:15.196241 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.196734 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.196738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.196740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.196832 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.197161 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.197164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.197166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.197364 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:15.197454 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.197975 (+ 521us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.198116 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:20:15.199297 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.186534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40375 (local address 127.29.3.65:44763)
0811 22:20:15.187035 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:15.187093 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:15.188414 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:15.188939 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:15.188946 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:15.189312 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:15.189825 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.189840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.190684 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.190687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.191571 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.191581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.192085 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.192089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.192268 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.192817 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:15.192834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:15.194629 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:15.196357 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:15.196364 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:15.196381 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:15.196634 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:15.196918 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:15.196922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:15.196924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:15.197068 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:15.197469 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:15.197477 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:15.197736 (+ 259us) client_negotiation.cc:769] Sending connection context
0811 22:20:15.197911 (+ 175us) client_negotiation.cc:240] Negotiation successful
0811 22:20:15.198271 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":136,"threads_started":1}
I20260811 22:20:15.200636 31171 heartbeater.cc:347] Connected to a master server at 127.29.3.126:40375
I20260811 22:20:15.200927 31171 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:15.201431 31171 heartbeater.cc:511] Master 127.29.3.126:40375 requested a full tablet report, sending...
I20260811 22:20:15.203145 30786 ts_manager.cc:194] Registered new tserver with Master: 48f3ec4f772b474387907140d8a5a5a6 (127.29.3.65:39157)
I20260811 22:20:15.204775 30786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:44763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:15.217233 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40375
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:15.338196 31187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:15.338505 31187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:15.338567 31187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:15.342625 31187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:15.342690 31187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:15.342770 31187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:15.348268 31187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:15.349404 31187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:15.350503 31187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:15.360208 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:15.360281 31187 server_base.cc:1034] running on GCE node
W20260811 22:20:15.360339 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:15.360203 31198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:15.360899 31187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:15.361532 31187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:15.362680 31187 hybrid_clock.cc:648] HybridClock initialized: now 1786486815362653 us; error 39 us; skew 500 ppm
Aug 11 22:20:15 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:15.365463 31187 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:15.366413 31187 webserver.cc:460] Webserver started at http://127.29.3.66:44279/ using document root <none> and password file <none>
I20260811 22:20:15.367036 31187 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:15.367102 31187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:15.367302 31187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:15.369628 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "550490fd107699272d7eee98402a4850"
server_key_iv: "e4cef426bca0e1f411c0c87eaf7b08ad"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.370271 31187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "550490fd107699272d7eee98402a4850"
server_key_iv: "e4cef426bca0e1f411c0c87eaf7b08ad"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.374269 31187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260811 22:20:15.376945 31207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.378166 31187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:15.378285 31187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "550490fd107699272d7eee98402a4850"
server_key_iv: "e4cef426bca0e1f411c0c87eaf7b08ad"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.378418 31187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:15.392792 31187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:15.396214 31187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:15.396430 31187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:15.397143 31187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:15.398018 31187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:15.398069 31187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.398124 31187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:15.398154 31187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.409312 31187 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:44391
I20260811 22:20:15.409333 31423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:44391 every 8 connection(s)
I20260811 22:20:15.410194 31187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:15.414413 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31187
I20260811 22:20:15.414522 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:15.414774 29709 external_mini_cluster.cc:1428] Setting key 7f2ebad73a5cb30d0754c4b26a00627a
Aug 11 22:20:15 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:15.424723 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.412155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:50615 (local address 127.29.3.126:40375)
0811 22:20:15.412290 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.412294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.413179 (+ 885us) server_negotiation.cc:407] Connection header received
0811 22:20:15.414531 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.414535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.414597 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.414671 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:15.416015 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.416740 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.417889 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.418223 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.418352 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.421051 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:15.421054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:15.421056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:15.421083 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:15.422519 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.423120 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.423122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.423123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.423192 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.423555 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.423557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.423558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.423747 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:15.423830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.424464 (+ 634us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.424580 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:20:15.425978 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.412522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40375 (local address 127.29.3.66:50615)
0811 22:20:15.413024 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:15.413067 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:15.414285 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:15.414863 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:15.414870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:15.415318 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:15.415890 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.415905 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.416870 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.416873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.417767 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.417776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.418515 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.418518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.418669 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.419245 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:15.419262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:15.420855 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:15.422635 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:15.422642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:15.422661 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:15.423030 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:15.423272 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:15.423275 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:15.423277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:15.423473 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:15.423843 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:15.423850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:15.424203 (+ 353us) client_negotiation.cc:769] Sending connection context
0811 22:20:15.424435 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 22:20:15.424832 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260811 22:20:15.427161 31424 heartbeater.cc:347] Connected to a master server at 127.29.3.126:40375
I20260811 22:20:15.427413 31424 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:15.427951 31424 heartbeater.cc:511] Master 127.29.3.126:40375 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
I20260811 22:20:15.429088 30786 ts_manager.cc:194] Registered new tserver with Master: 6dcb5f5c6fcb4d55b7b23f00697c13e8 (127.29.3.66:44391)
I20260811 22:20:15.429775 30786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:50615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:15.445454 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40375
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:35363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:15.558915 31434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:15.559154 31434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:15.559196 31434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:15.563354 31434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:15.563419 31434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:15.563505 31434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:15.569168 31434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:35363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40375
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:15.570185 31434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:15.571197 31434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:15.580838 31439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:15.580834 31440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:15.580920 31434 server_base.cc:1034] running on GCE node
W20260811 22:20:15.580830 31442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:15.581467 31434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:15.582120 31434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:15.583277 31434 hybrid_clock.cc:648] HybridClock initialized: now 1786486815583261 us; error 65 us; skew 500 ppm
Aug 11 22:20:15 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:15.585924 31434 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:15.586822 31434 webserver.cc:460] Webserver started at http://127.29.3.67:33153/ using document root <none> and password file <none>
I20260811 22:20:15.587363 31434 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:15.587424 31434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:15.587630 31434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:15.589674 31434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "a3670a60ec994534bdc70bd1add63964"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "1f4135fd88c7b4951eb44fa00d7f932a"
server_key_iv: "57aeb50d64acdfb1b8f56f378c32f118"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.590215 31434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "a3670a60ec994534bdc70bd1add63964"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "1f4135fd88c7b4951eb44fa00d7f932a"
server_key_iv: "57aeb50d64acdfb1b8f56f378c32f118"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.593904 31434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:15.596326 31456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.597489 31434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:15.597594 31434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "a3670a60ec994534bdc70bd1add63964"
format_stamp: "Formatted at 2026-08-11 22:20:15 on dist-test-slave-4tdj"
server_key: "1f4135fd88c7b4951eb44fa00d7f932a"
server_key_iv: "57aeb50d64acdfb1b8f56f378c32f118"
server_key_version: "encryptionkey@0"
I20260811 22:20:15.597694 31434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:15.616456 31434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:15.619472 31434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:15.619658 31434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:15.620290 31434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:15.621173 31434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:15.621224 31434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.621271 31434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:15.621300 31434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:15.630131 31434 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:38715
I20260811 22:20:15.630153 31686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:38715 every 8 connection(s)
I20260811 22:20:15.631078 31434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:15.632786 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31434
I20260811 22:20:15.632901 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:15.633193 29709 external_mini_cluster.cc:1428] Setting key 356b1fd7a2ed9ebf349e658a2755b900
Aug 11 22:20:15 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486815, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:15.644017 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.632820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:56053 (local address 127.29.3.126:40375)
0811 22:20:15.632966 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.632971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.634059 (+ 1088us) server_negotiation.cc:407] Connection header received
0811 22:20:15.635691 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.635694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.635753 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.635817 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:15.636940 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.637459 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.638576 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.638798 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.638870 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.641135 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:15.641139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:15.641141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:15.641165 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:15.642277 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.642764 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.642767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.642768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.642839 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.643090 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.643092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.643093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.643266 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:15.643340 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.643783 (+ 443us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.643887 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:20:15.644899 31695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.633363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40375 (local address 127.29.3.67:56053)
0811 22:20:15.633908 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:15.633967 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:15.635473 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:15.635968 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:15.635975 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:15.636328 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:15.636828 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.636843 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.637545 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.637547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.638486 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.638497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.638907 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.638909 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.639047 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.639592 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:15.639609 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:15.640990 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:15.642422 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:15.642427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:15.642440 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:15.642686 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:15.642901 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:15.642903 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:15.642904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:15.643026 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:15.643339 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:15.643344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:15.643582 (+ 238us) client_negotiation.cc:769] Sending connection context
0811 22:20:15.643752 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 22:20:15.644006 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":142,"threads_started":1}
I20260811 22:20:15.646049 31688 heartbeater.cc:347] Connected to a master server at 127.29.3.126:40375
I20260811 22:20:15.646320 31688 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:15.646878 31688 heartbeater.cc:511] Master 127.29.3.126:40375 requested a full tablet report, sending...
I20260811 22:20:15.647902 30786 ts_manager.cc:194] Registered new tserver with Master: a3670a60ec994534bdc70bd1add63964 (127.29.3.67:38715)
I20260811 22:20:15.648458 30786 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:56053
I20260811 22:20:15.659051 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:20:15.669377 30871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.662727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.29.3.126:40375)
0811 22:20:15.662845 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.662848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.662985 (+ 137us) server_negotiation.cc:407] Connection header received
0811 22:20:15.663094 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.663096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.663137 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.663202 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:15.663837 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.664347 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.665251 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.665478 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.665564 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.666941 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:15.666945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:15.666947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:15.666970 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:15.668066 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.668418 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.668420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.668421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.668486 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:15.668663 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:15.668665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:15.668666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:15.668882 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:15.668984 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.669160 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.669255 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:20:15.674221 30786 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:15.677268 30786 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:15.691921 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.686022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:39157 (local address 127.0.0.1:42318)
0811 22:20:15.686425 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:15.686483 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:15.686621 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:15.687447 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:15.687454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:15.687478 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:15.687638 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.687646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.689082 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.689085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.690349 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.690360 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.691351 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.691355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.691473 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.691524 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 22:20:15.691629 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 22:20:15.691760 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
I20260811 22:20:15.692090 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.686178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42318 (local address 127.29.3.65:39157)
0811 22:20:15.686608 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.686626 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.686652 (+ 26us) server_negotiation.cc:407] Connection header received
0811 22:20:15.686845 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.686851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.687078 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.687235 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:15.687773 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.688951 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.690470 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.691208 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.691334 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.691489 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.691837 (+ 348us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.691936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":121,"threads_started":1}
I20260811 22:20:15.692205 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.687260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.29.3.67:38715)
0811 22:20:15.687673 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.687681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.687701 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:20:15.687839 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.687844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.687988 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.688117 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:15.688370 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.689191 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.690745 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.691493 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.691590 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.691735 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.691981 (+ 246us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.692068 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":159,"threads_started":1}
I20260811 22:20:15.692431 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.686881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55280 (local address 127.29.3.66:44391)
0811 22:20:15.687896 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:15.687907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:15.687926 (+ 19us) server_negotiation.cc:407] Connection header received
0811 22:20:15.687995 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:15.688000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:15.688194 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:15.688344 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:15.688563 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.689491 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.690865 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.691449 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.691549 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.691698 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:15.692194 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 22:20:15.692289 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":117,"threads_started":1}
I20260811 22:20:15.692103 31719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.686273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:44391 (local address 127.0.0.1:55280)
0811 22:20:15.687283 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:15.687297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:15.687388 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:15.688336 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:15.688339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:15.688358 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:15.688450 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.688455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.689633 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.689636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.690763 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.690770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.691760 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.691763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.691843 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.691864 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:20:15.691928 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:20:15.691993 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":86,"threads_started":1}
I20260811 22:20:15.692473 31718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:15.686836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:38715 (local address 127.0.0.1:50540)
0811 22:20:15.687606 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:15.687619 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:15.687696 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:15.688111 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:15.688114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:15.688131 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:15.688243 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.688250 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.689339 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.689342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.690602 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:15.690614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:15.691593 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:15.691597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:15.691685 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:15.691708 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:20:15.691795 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 22:20:15.691872 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260811 22:20:15.695297 31045 tablet_service.cc:1467] Processing CreateTablet for tablet c91d4600e46c4a2596e61c3153d9c884 (DEFAULT_TABLE table=test-table [id=3f7980eb7601463994d1127bba2233f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:15.695300 31294 tablet_service.cc:1467] Processing CreateTablet for tablet c91d4600e46c4a2596e61c3153d9c884 (DEFAULT_TABLE table=test-table [id=3f7980eb7601463994d1127bba2233f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:15.696259 31045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91d4600e46c4a2596e61c3153d9c884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:15.696259 31294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91d4600e46c4a2596e61c3153d9c884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:15.699035 31558 tablet_service.cc:1467] Processing CreateTablet for tablet c91d4600e46c4a2596e61c3153d9c884 (DEFAULT_TABLE table=test-table [id=3f7980eb7601463994d1127bba2233f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:15.700201 31558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91d4600e46c4a2596e61c3153d9c884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:15.702881 31724 tablet_bootstrap.cc:492] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: Bootstrap starting.
I20260811 22:20:15.704239 31729 tablet_bootstrap.cc:492] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: Bootstrap starting.
I20260811 22:20:15.705375 31724 tablet_bootstrap.cc:654] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:15.706108 31724 log.cc:824] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:15.706456 31729 tablet_bootstrap.cc:654] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:15.707394 31729 log.cc:824] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:15.707545 31733 tablet_bootstrap.cc:492] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: Bootstrap starting.
I20260811 22:20:15.709486 31729 tablet_bootstrap.cc:492] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: No bootstrap required, opened a new log
I20260811 22:20:15.709702 31729 ts_tablet_manager.cc:1397] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260811 22:20:15.710095 31724 tablet_bootstrap.cc:492] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: No bootstrap required, opened a new log
I20260811 22:20:15.710372 31724 ts_tablet_manager.cc:1397] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260811 22:20:15.710381 31733 tablet_bootstrap.cc:654] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:15.714100 31733 log.cc:824] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:15.714493 31729 raft_consensus.cc:348] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:15.714924 31729 raft_consensus.cc:374] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:15.715032 31729 raft_consensus.cc:729] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f3ec4f772b474387907140d8a5a5a6, State: Initialized, Role: FOLLOWER
I20260811 22:20:15.715618 31729 consensus_queue.cc:260] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:15.715772 31724 raft_consensus.cc:348] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:15.716081 31724 raft_consensus.cc:374] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:15.716172 31724 raft_consensus.cc:729] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dcb5f5c6fcb4d55b7b23f00697c13e8, State: Initialized, Role: FOLLOWER
I20260811 22:20:15.716543 31733 tablet_bootstrap.cc:492] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: No bootstrap required, opened a new log
I20260811 22:20:15.716674 31724 consensus_queue.cc:260] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:15.716758 31733 ts_tablet_manager.cc:1397] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260811 22:20:15.716756 31729 ts_tablet_manager.cc:1428] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260811 22:20:15.717150 31171 heartbeater.cc:503] Master 127.29.3.126:40375 was elected leader, sending a full tablet report...
I20260811 22:20:15.717617 31724 ts_tablet_manager.cc:1428] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:20:15.718521 31424 heartbeater.cc:503] Master 127.29.3.126:40375 was elected leader, sending a full tablet report...
I20260811 22:20:15.721489 31733 raft_consensus.cc:348] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:15.721787 31733 raft_consensus.cc:374] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:15.721877 31733 raft_consensus.cc:729] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3670a60ec994534bdc70bd1add63964, State: Initialized, Role: FOLLOWER
I20260811 22:20:15.722368 31733 consensus_queue.cc:260] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:15.723481 31688 heartbeater.cc:503] Master 127.29.3.126:40375 was elected leader, sending a full tablet report...
I20260811 22:20:15.723632 31733 ts_tablet_manager.cc:1428] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260811 22:20:15.882007 31694 tablet.cc:2369] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:15.911448 31428 tablet.cc:2369] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:15.935540 31173 tablet.cc:2369] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:16.001348 31737 raft_consensus.cc:482] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:16.001588 31737 raft_consensus.cc:504] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:16.002889 31737 leader_election.cc:288] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3670a60ec994534bdc70bd1add63964 (127.29.3.67:38715), 6dcb5f5c6fcb4d55b7b23f00697c13e8 (127.29.3.66:44391)
I20260811 22:20:16.007084 31720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.003382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:33963 (local address 127.29.3.66:44391)
0811 22:20:16.003518 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:16.003523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:16.003695 (+ 172us) server_negotiation.cc:407] Connection header received
0811 22:20:16.003844 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:16.003848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:16.003890 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:16.003951 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:16.004391 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.004890 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.006086 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.006528 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.006625 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.006686 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:16.006904 (+ 218us) server_negotiation.cc:299] Negotiation successful
0811 22:20:16.006971 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 22:20:16.007085 31722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.003251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:42421 (local address 127.29.3.67:38715)
0811 22:20:16.003702 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:16.003706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:16.003834 (+ 128us) server_negotiation.cc:407] Connection header received
0811 22:20:16.003967 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:16.003971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:16.004026 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:16.004103 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:16.004381 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.004909 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.006115 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.006522 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.006607 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.006671 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:16.006905 (+ 234us) server_negotiation.cc:299] Negotiation successful
0811 22:20:16.006971 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260811 22:20:16.007110 31746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.003146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:38715 (local address 127.29.3.65:42421)
0811 22:20:16.003717 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:16.003730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:16.003854 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:16.004110 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:16.004113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:16.004127 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:16.004245 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:16.004253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.005025 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.005028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:16.005984 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:16.005994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.006629 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.006633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:16.006740 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.006802 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 22:20:16.006866 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:20:16.006940 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":350,"thread_start_us":126,"threads_started":1}
I20260811 22:20:16.007118 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.003147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:44391 (local address 127.29.3.65:33963)
0811 22:20:16.003596 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:16.003611 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:16.003741 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:16.003997 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:16.004000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:16.004050 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:16.004265 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:16.004273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.005007 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.005010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:16.005967 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:16.005977 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.006649 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.006653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:16.006771 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.006810 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:20:16.006878 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 22:20:16.006949 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":122,"thread_start_us":110,"threads_started":1}
I20260811 22:20:16.007855 31335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91d4600e46c4a2596e61c3153d9c884" candidate_uuid: "48f3ec4f772b474387907140d8a5a5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" is_pre_election: true
I20260811 22:20:16.007853 31598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91d4600e46c4a2596e61c3153d9c884" candidate_uuid: "48f3ec4f772b474387907140d8a5a5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3670a60ec994534bdc70bd1add63964" is_pre_election: true
I20260811 22:20:16.008186 31335 raft_consensus.cc:2454] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f3ec4f772b474387907140d8a5a5a6 in term 0.
I20260811 22:20:16.008186 31598 raft_consensus.cc:2454] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48f3ec4f772b474387907140d8a5a5a6 in term 0.
I20260811 22:20:16.008668 30952 leader_election.cc:302] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f3ec4f772b474387907140d8a5a5a6, 6dcb5f5c6fcb4d55b7b23f00697c13e8; no voters:
I20260811 22:20:16.008917 31737 raft_consensus.cc:2781] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:16.009004 31737 raft_consensus.cc:482] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:16.009052 31737 raft_consensus.cc:3037] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:16.010053 31737 raft_consensus.cc:504] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:16.010412 31737 leader_election.cc:288] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [CANDIDATE]: Term 1 election: Requested vote from peers a3670a60ec994534bdc70bd1add63964 (127.29.3.67:38715), 6dcb5f5c6fcb4d55b7b23f00697c13e8 (127.29.3.66:44391)
I20260811 22:20:16.010736 31598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91d4600e46c4a2596e61c3153d9c884" candidate_uuid: "48f3ec4f772b474387907140d8a5a5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3670a60ec994534bdc70bd1add63964"
I20260811 22:20:16.010780 31335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c91d4600e46c4a2596e61c3153d9c884" candidate_uuid: "48f3ec4f772b474387907140d8a5a5a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8"
I20260811 22:20:16.010874 31598 raft_consensus.cc:3037] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:16.010913 31335 raft_consensus.cc:3037] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:16.011893 31598 raft_consensus.cc:2454] T c91d4600e46c4a2596e61c3153d9c884 P a3670a60ec994534bdc70bd1add63964 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f3ec4f772b474387907140d8a5a5a6 in term 1.
I20260811 22:20:16.011929 31335 raft_consensus.cc:2454] T c91d4600e46c4a2596e61c3153d9c884 P 6dcb5f5c6fcb4d55b7b23f00697c13e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48f3ec4f772b474387907140d8a5a5a6 in term 1.
I20260811 22:20:16.012197 30954 leader_election.cc:302] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48f3ec4f772b474387907140d8a5a5a6, a3670a60ec994534bdc70bd1add63964; no voters:
I20260811 22:20:16.012367 31737 raft_consensus.cc:2781] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:16.012588 31737 raft_consensus.cc:686] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [term 1 LEADER]: Becoming Leader. State: Replica: 48f3ec4f772b474387907140d8a5a5a6, State: Running, Role: LEADER
I20260811 22:20:16.012905 31737 consensus_queue.cc:237] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } }
I20260811 22:20:16.015959 30785 catalog_manager.cc:5495] T c91d4600e46c4a2596e61c3153d9c884 P 48f3ec4f772b474387907140d8a5a5a6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48f3ec4f772b474387907140d8a5a5a6 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "48f3ec4f772b474387907140d8a5a5a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48f3ec4f772b474387907140d8a5a5a6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 39157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3670a60ec994534bdc70bd1add63964" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 38715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dcb5f5c6fcb4d55b7b23f00697c13e8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44391 } health_report { overall_health: UNKNOWN } } }
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:16.063508 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.047895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.29.3.65:39157)
0811 22:20:16.048066 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:16.048070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:16.048139 (+ 69us) server_negotiation.cc:407] Connection header received
0811 22:20:16.048247 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:16.048249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:16.048293 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:16.048361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:16.048989 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.049478 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.050224 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.050446 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.050516 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.060408 (+ 9892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:16.060424 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:16.060443 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:16.060471 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:16.061932 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:16.062366 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:16.062372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:16.062373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:16.062458 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:16.062682 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:16.062684 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:16.062685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:16.062948 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:16.063103 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:16.063240 (+ 137us) server_negotiation.cc:299] Negotiation successful
0811 22:20:16.063345 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:16.090902 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.083063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.29.3.65:39157)
0811 22:20:16.083201 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:16.083205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:16.083371 (+ 166us) server_negotiation.cc:407] Connection header received
0811 22:20:16.083526 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:16.083529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:16.083579 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:16.083643 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:16.084343 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.084863 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.085694 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.085929 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.086016 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.088205 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:16.088209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:16.088212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:16.088239 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:16.089290 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:16.089692 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:16.089696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:16.089697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:16.089774 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:16.090021 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:16.090024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:16.090026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:16.090194 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:16.090354 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:16.090619 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 22:20:16.090739 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[30695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[30695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:16.116205 31717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:16.108469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.29.3.65:39157)
0811 22:20:16.108602 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:16.108606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:16.108749 (+ 143us) server_negotiation.cc:407] Connection header received
0811 22:20:16.108882 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:16.108885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:16.108927 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:16.108994 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:16.109670 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.110158 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.111032 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:16.111251 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:16.111338 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:16.113564 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:16.113568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:16.113570 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:16.113597 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:16.114718 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:16.115093 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:16.115096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:16.115097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:16.115173 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:16.115414 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:16.115417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:16.115419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:16.115582 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:16.115723 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:16.115938 (+ 215us) server_negotiation.cc:299] Negotiation successful
0811 22:20:16.116051 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260811 22:20:16.117825 31005 tablet_service.cc:2908] Rejecting scan request for tablet c91d4600e46c4a2596e61c3153d9c884: Uninitialized: clean time has not yet been initialized
I20260811 22:20:16.119846 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30916
I20260811 22:20:16.140828 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31187
I20260811 22:20:16.151446 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31434
I20260811 22:20:16.167409 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30723
2026-08-11T22:20:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2747 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[31766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[31766](info): set up 2 sockets
Aug 11 22:20:16 dist-test-slave-4tdj krb5kdc[31766](info): commencing operation
krb5kdc: starting...
W20260811 22:20:17.347376 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.153s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:17 dist-test-slave-4tdj krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:17.469192 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42539
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:17.585044 31791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:17.585306 31791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:17.585346 31791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:17.589258 31791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:17.589318 31791 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 22:20:17.589334 31791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:17.589350 31791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:17.589370 31791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:17.589385 31791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:17.594695 31791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:42539
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:17.595721 31791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:17.596690 31791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:17.605150 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:17.605338 31811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:17.605275 31791 server_base.cc:1034] running on GCE node
W20260811 22:20:17.605311 31813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:17.605988 31791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:17.607092 31791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:17.608261 31791 hybrid_clock.cc:648] HybridClock initialized: now 1786486817608246 us; error 27 us; skew 500 ppm
Aug 11 22:20:17 dist-test-slave-4tdj krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:17.610962 31791 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:17.611871 31791 webserver.cc:460] Webserver started at http://127.29.3.126:33315/ using document root <none> and password file <none>
I20260811 22:20:17.612424 31791 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:17.612488 31791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:17.612696 31791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:17.614830 31791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "1eea6086b851460a885e657bdbda7af3"
format_stamp: "Formatted at 2026-08-11 22:20:17 on dist-test-slave-4tdj"
server_key: "4840c2ea2c8e42a0cc08acdf0a166d15"
server_key_iv: "4ac408dd2a2bf37fd3c6616324967c64"
server_key_version: "encryptionkey@0"
I20260811 22:20:17.615386 31791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "1eea6086b851460a885e657bdbda7af3"
format_stamp: "Formatted at 2026-08-11 22:20:17 on dist-test-slave-4tdj"
server_key: "4840c2ea2c8e42a0cc08acdf0a166d15"
server_key_iv: "4ac408dd2a2bf37fd3c6616324967c64"
server_key_version: "encryptionkey@0"
I20260811 22:20:17.619278 31791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:20:17.621682 31825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:17.622963 31791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:17.623087 31791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "1eea6086b851460a885e657bdbda7af3"
format_stamp: "Formatted at 2026-08-11 22:20:17 on dist-test-slave-4tdj"
server_key: "4840c2ea2c8e42a0cc08acdf0a166d15"
server_key_iv: "4ac408dd2a2bf37fd3c6616324967c64"
server_key_version: "encryptionkey@0"
I20260811 22:20:17.623190 31791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:17.639472 31791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:17.642756 31791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:17.642946 31791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:17.650838 31791 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:42539
I20260811 22:20:17.650897 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:42539 every 8 connection(s)
I20260811 22:20:17.651798 31791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:17.654562 31932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:17.655046 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31791
I20260811 22:20:17.655179 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:17.655463 29709 external_mini_cluster.cc:1428] Setting key 626ae8c006a4688ae62286f5203c473f
I20260811 22:20:17.661705 31932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: Bootstrap starting.
Aug 11 22:20:17 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:17.664388 31932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:17.665244 31932 log.cc:824] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:17.667135 31932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: No bootstrap required, opened a new log
I20260811 22:20:17.670208 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:17.656849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.29.3.126:42539)
0811 22:20:17.657245 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:17.657260 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:17.657297 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:20:17.657977 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:17.657995 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:17.658369 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:17.658652 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:17.659521 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.660488 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.661692 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.662189 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.662349 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:17.664995 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:17.665000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:17.665013 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:17.665042 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:17.667503 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:17.667965 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:17.667969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:17.667974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:17.668079 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:17.668337 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:17.668339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:17.668340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:17.668711 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:17.668862 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:17.669120 (+ 258us) server_negotiation.cc:299] Negotiation successful
0811 22:20:17.669412 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260811 22:20:17.670617 31932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eea6086b851460a885e657bdbda7af3" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42539 } }
I20260811 22:20:17.670836 31932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:17.670892 31932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eea6086b851460a885e657bdbda7af3, State: Initialized, Role: FOLLOWER
I20260811 22:20:17.671324 31932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eea6086b851460a885e657bdbda7af3" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42539 } }
I20260811 22:20:17.671456 31932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:17.671545 31932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:17.671651 31932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:17.672514 31932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eea6086b851460a885e657bdbda7af3" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42539 } }
I20260811 22:20:17.672928 31932 leader_election.cc:302] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eea6086b851460a885e657bdbda7af3; no voters:
I20260811 22:20:17.673287 31932 leader_election.cc:288] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:17.673381 31939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:17.673584 31939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [term 1 LEADER]: Becoming Leader. State: Replica: 1eea6086b851460a885e657bdbda7af3, State: Running, Role: LEADER
I20260811 22:20:17.673969 31939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eea6086b851460a885e657bdbda7af3" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42539 } }
I20260811 22:20:17.674542 31932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:17.675631 31940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eea6086b851460a885e657bdbda7af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eea6086b851460a885e657bdbda7af3" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42539 } } }
I20260811 22:20:17.675767 31940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:17.676077 31942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eea6086b851460a885e657bdbda7af3. Latest consensus state: current_term: 1 leader_uuid: "1eea6086b851460a885e657bdbda7af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eea6086b851460a885e657bdbda7af3" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42539 } } }
I20260811 22:20:17.676172 31942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:17.677016 31952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:17.679633 31952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:17.685006 31952 catalog_manager.cc:1347] Generated new cluster ID: 72b81a5cf6f843118aaa5f2e560b812b
I20260811 22:20:17.685074 31952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:17.697479 31952 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:17.698510 31952 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:17.709894 31952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: Generated new TSK 0
I20260811 22:20:17.710850 31952 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:17.749749 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42539
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:17.861197 31971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:17.861436 31971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:17.861479 31971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:17.865353 31971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:17.865404 31971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:17.865442 31971 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:17.865476 31971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:17.870638 31971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:17.871583 31971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:17.872607 31971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:17.882156 31985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:17.882184 31988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:17.882171 31984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:17.882557 31971 server_base.cc:1034] running on GCE node
I20260811 22:20:17.882963 31971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:17.883565 31971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:17.884733 31971 hybrid_clock.cc:648] HybridClock initialized: now 1786486817884709 us; error 35 us; skew 500 ppm
Aug 11 22:20:17 dist-test-slave-4tdj krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:17.887778 31971 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:17.888684 31971 webserver.cc:460] Webserver started at http://127.29.3.65:43813/ using document root <none> and password file <none>
I20260811 22:20:17.889213 31971 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:17.889277 31971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:17.889456 31971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:17.891551 31971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7"
format_stamp: "Formatted at 2026-08-11 22:20:17 on dist-test-slave-4tdj"
server_key: "20186b8f9a6123e618e4175258933ee2"
server_key_iv: "6e5c9ce2b92bbdacbe22b107807bbc38"
server_key_version: "encryptionkey@0"
I20260811 22:20:17.892102 31971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7"
format_stamp: "Formatted at 2026-08-11 22:20:17 on dist-test-slave-4tdj"
server_key: "20186b8f9a6123e618e4175258933ee2"
server_key_iv: "6e5c9ce2b92bbdacbe22b107807bbc38"
server_key_version: "encryptionkey@0"
I20260811 22:20:17.896092 31971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:17.898594 32001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:17.899837 31971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:17.899977 31971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7"
format_stamp: "Formatted at 2026-08-11 22:20:17 on dist-test-slave-4tdj"
server_key: "20186b8f9a6123e618e4175258933ee2"
server_key_iv: "6e5c9ce2b92bbdacbe22b107807bbc38"
server_key_version: "encryptionkey@0"
I20260811 22:20:17.900082 31971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:17.912163 31971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:17.915359 31971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:17.915560 31971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:17.917245 31971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:17.917302 31971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:17.917428 31971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:17.917460 31971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:17 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:17.928236 31971 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:36387
I20260811 22:20:17.928306 32216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:36387 every 8 connection(s)
I20260811 22:20:17.929183 31971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:17.931761 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:17.918036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:51885 (local address 127.29.3.126:42539)
0811 22:20:17.918198 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:17.918202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:17.919485 (+ 1283us) server_negotiation.cc:407] Connection header received
0811 22:20:17.920489 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:17.920493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:17.920558 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:17.920672 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:17.921853 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.922418 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.923506 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.923748 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.923838 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:17.928317 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:17.928322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:17.928324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:17.928357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:17.929719 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:17.930281 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:17.930284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:17.930285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:17.930436 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:17.930783 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:17.930786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:17.930787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:17.931008 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:17.931095 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:17.931529 (+ 434us) server_negotiation.cc:299] Negotiation successful
0811 22:20:17.931638 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:20:17.932524 32032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:17.918873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.29.3.65:51885)
0811 22:20:17.919317 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:17.919366 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:17.920284 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:17.920852 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:17.920859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:17.921207 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:17.921705 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:17.921719 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.922535 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.922539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:17.923387 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:17.923397 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.923899 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.923904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:17.924084 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:17.924582 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:17.924604 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:17.928101 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:17.929854 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:17.929860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:17.929875 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:17.930193 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:17.930524 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:17.930526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:17.930527 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:17.930659 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:17.931092 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:17.931097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:17.931337 (+ 240us) client_negotiation.cc:769] Sending connection context
0811 22:20:17.931489 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 22:20:17.931740 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260811 22:20:17.933842 32219 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42539
I20260811 22:20:17.934005 32219 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:17.934532 32219 heartbeater.cc:511] Master 127.29.3.126:42539 requested a full tablet report, sending...
I20260811 22:20:17.936064 31859 ts_manager.cc:194] Registered new tserver with Master: 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387)
I20260811 22:20:17.936784 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31971
I20260811 22:20:17.936898 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:17.937177 29709 external_mini_cluster.cc:1428] Setting key 0a3241a5b04b09cc32ce3d7872b914c8
I20260811 22:20:17.937536 31859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:51885
I20260811 22:20:17.942801 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:17.936159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38491 (local address 127.29.3.126:42539)
0811 22:20:17.936281 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:17.936285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:17.936489 (+ 204us) server_negotiation.cc:407] Connection header received
0811 22:20:17.936598 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:17.936600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:17.936642 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:17.936704 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:17.937158 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.937722 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.938746 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.938963 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.939048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:17.940063 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:17.940067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:17.940070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:17.940100 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:17.941332 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:17.941717 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:17.941721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:17.941722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:17.941793 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:17.942036 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:17.942039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:17.942041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:17.942205 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:17.942312 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:17.942555 (+ 243us) server_negotiation.cc:299] Negotiation successful
0811 22:20:17.942656 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:20:17.942814 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:17.936042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.29.3.65:38491)
0811 22:20:17.936407 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:17.936419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:17.936511 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:17.936731 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:17.936734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:17.936943 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:17.937049 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:17.937054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.937861 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.937864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:17.938636 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:17.938645 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:17.939062 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:17.939065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:17.939156 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:17.939610 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:17.939623 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:17.939933 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:17.941431 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:17.941434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:17.941437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:17.941631 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:17.941874 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:17.941877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:17.941879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:17.941959 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:17.942315 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:17.942319 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:17.942406 (+ 87us) client_negotiation.cc:769] Sending connection context
0811 22:20:17.942491 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 22:20:17.942636 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:17.965178 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42539
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:18.077852 32234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:18.078109 32234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:18.078158 32234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:18.082329 32234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:18.082398 32234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:18.082441 32234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:18.082506 32234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:18.087884 32234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:18.089021 32234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:18.090060 32234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:18.099570 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:18.099553 32250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:18.099681 32234 server_base.cc:1034] running on GCE node
W20260811 22:20:18.099529 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:18.100226 32234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:18.100827 32234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:18.101974 32234 hybrid_clock.cc:648] HybridClock initialized: now 1786486818101947 us; error 38 us; skew 500 ppm
Aug 11 22:20:18 dist-test-slave-4tdj krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:18.104626 32234 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:18.105580 32234 webserver.cc:460] Webserver started at http://127.29.3.66:43219/ using document root <none> and password file <none>
I20260811 22:20:18.106215 32234 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:18.106284 32234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:18.106537 32234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:18.108762 32234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "057c5b6205ed419fa0fb4db6118f0802"
format_stamp: "Formatted at 2026-08-11 22:20:18 on dist-test-slave-4tdj"
server_key: "5230eddbfca6beb269022fac48b0233a"
server_key_iv: "ac0255fa9d4e2ce81f98189a98b519a0"
server_key_version: "encryptionkey@0"
I20260811 22:20:18.109325 32234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "057c5b6205ed419fa0fb4db6118f0802"
format_stamp: "Formatted at 2026-08-11 22:20:18 on dist-test-slave-4tdj"
server_key: "5230eddbfca6beb269022fac48b0233a"
server_key_iv: "ac0255fa9d4e2ce81f98189a98b519a0"
server_key_version: "encryptionkey@0"
I20260811 22:20:18.113181 32234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260811 22:20:18.115617 32257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:18.116802 32234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:18.116906 32234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "057c5b6205ed419fa0fb4db6118f0802"
format_stamp: "Formatted at 2026-08-11 22:20:18 on dist-test-slave-4tdj"
server_key: "5230eddbfca6beb269022fac48b0233a"
server_key_iv: "ac0255fa9d4e2ce81f98189a98b519a0"
server_key_version: "encryptionkey@0"
I20260811 22:20:18.117018 32234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:18.140697 32234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:18.143532 32234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:18.143715 32234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:18.145252 32234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:18.145308 32234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:18.145363 32234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:18.145418 32234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:18 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:18.155730 32234 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:38437
I20260811 22:20:18.155769 32488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:38437 every 8 connection(s)
I20260811 22:20:18.156876 32234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:18.158792 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.146025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:38617 (local address 127.29.3.126:42539)
0811 22:20:18.146190 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.146195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.147590 (+ 1395us) server_negotiation.cc:407] Connection header received
0811 22:20:18.148755 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.148759 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.148822 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.148895 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:18.150115 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.150698 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.151915 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.152238 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.152339 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.154969 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:18.154972 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:18.154974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:18.155002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:18.156487 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.157191 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.157194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.157196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.157313 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.157709 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.157712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.157714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.157935 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:18.158017 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.158546 (+ 529us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.158663 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:20:18.159555 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.146871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.29.3.66:38617)
0811 22:20:18.147432 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.147486 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.148505 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.149096 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.149105 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.149456 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:18.149960 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.149974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.150829 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.150832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.151793 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.151803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.152409 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.152413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.152585 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.153150 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:18.153167 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:18.154788 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:18.156655 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.156663 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.156676 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.157089 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.157426 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.157429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.157431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.157597 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.158038 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:18.158044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:18.158275 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.158481 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.158770 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":120,"threads_started":1}
I20260811 22:20:18.160748 32489 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42539
I20260811 22:20:18.160888 32489 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:18.161413 32489 heartbeater.cc:511] Master 127.29.3.126:42539 requested a full tablet report, sending...
I20260811 22:20:18.162443 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 32234
I20260811 22:20:18.162459 31859 ts_manager.cc:194] Registered new tserver with Master: 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:18.162593 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:18.162922 29709 external_mini_cluster.cc:1428] Setting key 781ac7f1d68c949843280586629a0910
I20260811 22:20:18.163084 31859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:38617
I20260811 22:20:18.170462 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.162948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:53789 (local address 127.29.3.126:42539)
0811 22:20:18.163408 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.163412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.163650 (+ 238us) server_negotiation.cc:407] Connection header received
0811 22:20:18.163782 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.163786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.163826 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.163899 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:18.164374 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.165062 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.165845 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.166139 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.166238 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.167348 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:18.167351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:18.167353 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:18.167375 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:18.168776 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.169226 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.169228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.169230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.169307 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.169581 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.169584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.169586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.169843 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:18.169924 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.170208 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.170343 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387}
I20260811 22:20:18.170742 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.163084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.29.3.66:53789)
0811 22:20:18.163573 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.163587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.163686 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.163934 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.163937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.164139 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:18.164249 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.164256 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.165190 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.165192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.165736 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.165744 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.166241 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.166244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.166353 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.166797 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:18.166811 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:18.167194 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:18.168887 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.168890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.168892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.169133 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.169390 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.169393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.169395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.169488 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.169945 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:18.169948 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:18.170051 (+ 103us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.170168 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.170360 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":105,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:18.191530 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42539
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:18.303220 32508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:18.303472 32508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:18.303532 32508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:18.307726 32508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:18.307777 32508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:18.307813 32508 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:18.307849 32508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:18.312865 32508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:18.313815 32508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:18.314795 32508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:18.324220 32521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:18.324280 32524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:18.324334 32508 server_base.cc:1034] running on GCE node
W20260811 22:20:18.324249 32522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:18.324862 32508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:18.325443 32508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:18.326612 32508 hybrid_clock.cc:648] HybridClock initialized: now 1786486818326596 us; error 28 us; skew 500 ppm
Aug 11 22:20:18 dist-test-slave-4tdj krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:18.329187 32508 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:18.329967 32508 webserver.cc:460] Webserver started at http://127.29.3.67:37897/ using document root <none> and password file <none>
I20260811 22:20:18.330530 32508 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:18.330593 32508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:18.330770 32508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:18.332685 32508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "9f95f5b1a99d46d5b1026a2aef01315b"
format_stamp: "Formatted at 2026-08-11 22:20:18 on dist-test-slave-4tdj"
server_key: "9563fd9106bac2514c6fefbdc5165ca1"
server_key_iv: "a545adeab2a18cac678ce138e64f8927"
server_key_version: "encryptionkey@0"
I20260811 22:20:18.333166 32508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "9f95f5b1a99d46d5b1026a2aef01315b"
format_stamp: "Formatted at 2026-08-11 22:20:18 on dist-test-slave-4tdj"
server_key: "9563fd9106bac2514c6fefbdc5165ca1"
server_key_iv: "a545adeab2a18cac678ce138e64f8927"
server_key_version: "encryptionkey@0"
I20260811 22:20:18.336807 32508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:20:18.339140 32539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:18.340298 32508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:18.340396 32508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "9f95f5b1a99d46d5b1026a2aef01315b"
format_stamp: "Formatted at 2026-08-11 22:20:18 on dist-test-slave-4tdj"
server_key: "9563fd9106bac2514c6fefbdc5165ca1"
server_key_iv: "a545adeab2a18cac678ce138e64f8927"
server_key_version: "encryptionkey@0"
I20260811 22:20:18.340494 32508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:18.351399 32508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:18.354007 32508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:18.354183 32508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:18.355639 32508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:18.355688 32508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:18.355728 32508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:18.355756 32508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:18 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:18.365587 32508 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:34605
I20260811 22:20:18.365855 32763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:34605 every 8 connection(s)
I20260811 22:20:18.366703 32508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:18.367746 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 32508
I20260811 22:20:18.367843 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:18.368131 29709 external_mini_cluster.cc:1428] Setting key bf49d7bb2c90e87b6645c597ef3c768b
I20260811 22:20:18.369143 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.356432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:37719 (local address 127.29.3.126:42539)
0811 22:20:18.356593 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.356598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.357483 (+ 885us) server_negotiation.cc:407] Connection header received
0811 22:20:18.358551 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.358554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.358613 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.358707 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:18.360019 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.360578 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.361837 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.362206 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.362363 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.365262 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:18.365266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:18.365269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:18.365299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:18.366879 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.367446 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.367449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.367451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.367549 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.367937 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.367940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.367942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.368201 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:18.368310 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.368862 (+ 552us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.368987 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:18.370133 32555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.356791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.29.3.67:37719)
0811 22:20:18.357259 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.357307 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.358252 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.358904 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.358913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.359294 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:18.359874 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.359889 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.360705 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.360709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.361717 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.361728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.362426 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.362431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.362641 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.363225 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:18.363242 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:18.365054 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:18.366985 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.366990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.367002 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.367319 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.367642 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.367645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.367648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.367791 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.368303 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:18.368308 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:18.368578 (+ 270us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.368810 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.369141 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":109,"threads_started":1}
I20260811 22:20:18.371307 32765 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42539
I20260811 22:20:18.371448 32765 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:18.371948 32765 heartbeater.cc:511] Master 127.29.3.126:42539 requested a full tablet report, sending...
I20260811 22:20:18.373330 31861 ts_manager.cc:194] Registered new tserver with Master: 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67:34605)
I20260811 22:20:18.373910 31861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:37719
I20260811 22:20:18.379523 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.373348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:38833 (local address 127.29.3.126:42539)
0811 22:20:18.373468 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.373472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.373698 (+ 226us) server_negotiation.cc:407] Connection header received
0811 22:20:18.373797 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.373800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.373843 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.373903 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:18.374356 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.375060 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.375938 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.376170 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.376245 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.377231 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:18.377234 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:18.377236 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:18.377253 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:18.378317 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.378665 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.378667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.378668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.378735 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.378922 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.378924 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.378925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.379080 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:18.379148 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.379331 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.379417 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:20:18.379529 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.373275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.29.3.67:38833)
0811 22:20:18.373618 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.373630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.373718 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.373929 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.373932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.374108 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:18.374232 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.374239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.375178 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.375181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.375814 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.375824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.376259 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.376261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.376366 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.376809 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:18.376820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:18.377124 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:18.378406 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.378409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.378410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.378589 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.378794 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:18.378796 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:18.378797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:18.378864 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:18.379140 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:18.379142 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:18.379216 (+ 74us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.379289 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.379387 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
I20260811 22:20:18.382321 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:20:18 dist-test-slave-4tdj krb5kdc[31766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:20:18 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:18.407351 31936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.399739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.29.3.126:42539)
0811 22:20:18.399878 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.399882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.399894 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:18.399942 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.399944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.399984 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.400059 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:18.400765 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.401288 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.402096 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.402362 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.402461 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.404783 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:18.404786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:18.404788 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:18.404816 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:18.405937 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.406315 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.406318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.406320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.406414 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:18.406679 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:18.406682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:18.406683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:18.406866 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:18.406965 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.407133 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.407229 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:20:18.410218 31861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:18.413275 31861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:18.427736 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.421903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.29.3.65:36387)
0811 22:20:18.422206 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.422225 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.422334 (+ 109us) server_negotiation.cc:407] Connection header received
0811 22:20:18.422579 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.422585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.422807 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.422958 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.423373 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.424479 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.426150 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.427028 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.427184 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.427376 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.427473 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.427573 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":104,"threads_started":1}
I20260811 22:20:18.427721 327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.422618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:34605 (local address 127.0.0.1:38000)
0811 22:20:18.422863 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.422877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.422971 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.424260 (+ 1289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.424263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.424281 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.424409 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.424416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.425379 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.425382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.426391 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.426403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.427209 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.427212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.427335 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.427353 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.427418 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.427544 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":23,"thread_start_us":100,"threads_started":1}
I20260811 22:20:18.427726 323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.421776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:36387 (local address 127.0.0.1:55564)
0811 22:20:18.422161 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.422219 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.422392 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.423093 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.423098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.423119 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.423248 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.423255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.424567 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.424570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.425987 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.425999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.427161 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.427166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.427285 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.427333 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.427500 (+ 167us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.427557 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":131,"threads_started":1}
I20260811 22:20:18.428529 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.422777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38000 (local address 127.29.3.67:34605)
0811 22:20:18.423835 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.423845 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.423868 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:20:18.423935 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.423940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.424120 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.424261 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.424525 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.425266 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.426524 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.427089 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.428114 (+ 1025us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.428282 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.428331 (+ 49us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.428419 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":109,"threads_started":1}
I20260811 22:20:18.428653 330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.422931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38437 (local address 127.0.0.1:34360)
0811 22:20:18.423476 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.423499 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.423582 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.424282 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.424285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.424298 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.424409 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.424416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.426010 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.426013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.427168 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.427181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.428291 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.428294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.428377 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.428401 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.428461 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.428524 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":3,"spinlock_wait_cycles":1792,"thread_start_us":83,"threads_started":1}
I20260811 22:20:18.429380 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.423438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.29.3.66:38437)
0811 22:20:18.423915 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.423923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.423941 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:20:18.423999 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.424003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.424145 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.424558 (+ 413us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.424689 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.425879 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.427315 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.428087 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.428202 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.429096 (+ 894us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.429148 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.429241 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":112,"threads_started":1}
I20260811 22:20:18.431156 32100 tablet_service.cc:1467] Processing CreateTablet for tablet aaf64f4eafc949099ab4d8028419ac28 (DEFAULT_TABLE table=test-table [id=50e016a589a34cae85278fad133b3f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:18.431337 32630 tablet_service.cc:1467] Processing CreateTablet for tablet aaf64f4eafc949099ab4d8028419ac28 (DEFAULT_TABLE table=test-table [id=50e016a589a34cae85278fad133b3f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:18.432156 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf64f4eafc949099ab4d8028419ac28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:18.431879 32350 tablet_service.cc:1467] Processing CreateTablet for tablet aaf64f4eafc949099ab4d8028419ac28 (DEFAULT_TABLE table=test-table [id=50e016a589a34cae85278fad133b3f17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:18.432397 32630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf64f4eafc949099ab4d8028419ac28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:18.432801 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaf64f4eafc949099ab4d8028419ac28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:18.438446 333 tablet_bootstrap.cc:492] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Bootstrap starting.
I20260811 22:20:18.440161 334 tablet_bootstrap.cc:492] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Bootstrap starting.
I20260811 22:20:18.442189 333 tablet_bootstrap.cc:654] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:18.442193 334 tablet_bootstrap.cc:654] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:18.442691 336 tablet_bootstrap.cc:492] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Bootstrap starting.
I20260811 22:20:18.443152 334 log.cc:824] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:18.443152 333 log.cc:824] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:18.444867 336 tablet_bootstrap.cc:654] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:18.445490 333 tablet_bootstrap.cc:492] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: No bootstrap required, opened a new log
I20260811 22:20:18.445490 334 tablet_bootstrap.cc:492] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: No bootstrap required, opened a new log
I20260811 22:20:18.445700 334 ts_tablet_manager.cc:1397] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:20:18.445700 333 ts_tablet_manager.cc:1397] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:20:18.445803 336 log.cc:824] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:18.447662 336 tablet_bootstrap.cc:492] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: No bootstrap required, opened a new log
I20260811 22:20:18.447853 336 ts_tablet_manager.cc:1397] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 22:20:18.449626 334 raft_consensus.cc:348] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.449626 333 raft_consensus.cc:348] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.449918 334 raft_consensus.cc:374] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:18.449918 333 raft_consensus.cc:374] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:18.449976 334 raft_consensus.cc:729] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057c5b6205ed419fa0fb4db6118f0802, State: Initialized, Role: FOLLOWER
I20260811 22:20:18.449983 333 raft_consensus.cc:729] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d74cec7d93f4f6e8c5a0da54b3266e7, State: Initialized, Role: FOLLOWER
I20260811 22:20:18.450492 334 consensus_queue.cc:260] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.450492 333 consensus_queue.cc:260] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.450922 336 raft_consensus.cc:348] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.451187 336 raft_consensus.cc:374] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:18.451237 336 raft_consensus.cc:729] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f95f5b1a99d46d5b1026a2aef01315b, State: Initialized, Role: FOLLOWER
I20260811 22:20:18.451241 32219 heartbeater.cc:503] Master 127.29.3.126:42539 was elected leader, sending a full tablet report...
I20260811 22:20:18.451551 333 ts_tablet_manager.cc:1428] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 22:20:18.451551 334 ts_tablet_manager.cc:1428] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:20:18.451706 336 consensus_queue.cc:260] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.451877 32489 heartbeater.cc:503] Master 127.29.3.126:42539 was elected leader, sending a full tablet report...
I20260811 22:20:18.452601 32765 heartbeater.cc:503] Master 127.29.3.126:42539 was elected leader, sending a full tablet report...
I20260811 22:20:18.453224 336 ts_tablet_manager.cc:1428] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 22:20:18.519994 343 raft_consensus.cc:482] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:18.520226 343 raft_consensus.cc:504] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.521633 343 leader_election.cc:288] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67:34605), 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:18.526410 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.521951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38033 (local address 127.29.3.67:34605)
0811 22:20:18.522074 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.522078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.522314 (+ 236us) server_negotiation.cc:407] Connection header received
0811 22:20:18.522378 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.522381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.522436 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.522512 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.522786 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.523552 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.525105 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.525724 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.525832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.525895 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.526174 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.526260 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:20:18.526398 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.522391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:53231 (local address 127.29.3.66:38437)
0811 22:20:18.522523 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.522527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.522643 (+ 116us) server_negotiation.cc:407] Connection header received
0811 22:20:18.522748 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.522751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.522802 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.522880 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.523082 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.523770 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.525103 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.525729 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.525832 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.525895 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.526174 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.526245 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:20:18.526453 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.521819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:34605 (local address 127.29.3.65:38033)
0811 22:20:18.522125 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.522141 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.522247 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.522530 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.522533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.522553 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.522681 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.522688 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.523650 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.523652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.524983 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.524995 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.525846 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.525850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.526030 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.526073 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.526143 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.526255 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":135,"thread_start_us":124,"threads_started":1}
I20260811 22:20:18.526453 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.522254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38437 (local address 127.29.3.65:53231)
0811 22:20:18.522554 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.522567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.522654 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.522887 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.522889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.522906 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.522989 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.522993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.523859 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.523862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.525000 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.525007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.525846 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.525850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.526026 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.526074 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.526148 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.526252 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":84,"thread_start_us":109,"threads_started":1}
I20260811 22:20:18.527139 32670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf64f4eafc949099ab4d8028419ac28" candidate_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" is_pre_election: true
I20260811 22:20:18.527290 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf64f4eafc949099ab4d8028419ac28" candidate_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057c5b6205ed419fa0fb4db6118f0802" is_pre_election: true
I20260811 22:20:18.527467 32670 raft_consensus.cc:2454] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d74cec7d93f4f6e8c5a0da54b3266e7 in term 0.
I20260811 22:20:18.527586 32389 raft_consensus.cc:2454] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d74cec7d93f4f6e8c5a0da54b3266e7 in term 0.
I20260811 22:20:18.527992 32002 leader_election.cc:302] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d74cec7d93f4f6e8c5a0da54b3266e7, 9f95f5b1a99d46d5b1026a2aef01315b; no voters:
I20260811 22:20:18.528251 343 raft_consensus.cc:2781] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:18.528328 343 raft_consensus.cc:482] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:18.528357 343 raft_consensus.cc:3037] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:18.529886 343 raft_consensus.cc:504] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.530282 343 leader_election.cc:288] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [CANDIDATE]: Term 1 election: Requested vote from peers 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67:34605), 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:18.530668 32670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf64f4eafc949099ab4d8028419ac28" candidate_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f95f5b1a99d46d5b1026a2aef01315b"
I20260811 22:20:18.530702 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaf64f4eafc949099ab4d8028419ac28" candidate_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057c5b6205ed419fa0fb4db6118f0802"
I20260811 22:20:18.530807 32389 raft_consensus.cc:3037] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:18.530807 32670 raft_consensus.cc:3037] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:18.532378 32670 raft_consensus.cc:2454] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d74cec7d93f4f6e8c5a0da54b3266e7 in term 1.
I20260811 22:20:18.532378 32389 raft_consensus.cc:2454] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d74cec7d93f4f6e8c5a0da54b3266e7 in term 1.
I20260811 22:20:18.532670 32002 leader_election.cc:302] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057c5b6205ed419fa0fb4db6118f0802, 8d74cec7d93f4f6e8c5a0da54b3266e7; no voters:
I20260811 22:20:18.532822 343 raft_consensus.cc:2781] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:18.533031 343 raft_consensus.cc:686] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 LEADER]: Becoming Leader. State: Replica: 8d74cec7d93f4f6e8c5a0da54b3266e7, State: Running, Role: LEADER
I20260811 22:20:18.533365 343 consensus_queue.cc:237] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } }
I20260811 22:20:18.536518 31861 catalog_manager.cc:5495] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:18.557188 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.553758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55570 (local address 127.29.3.65:36387)
0811 22:20:18.553886 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.553890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.553903 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:18.554085 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.554088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.554145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.554226 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:18.554465 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.554977 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.555794 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.556016 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.556094 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.556422 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:18.556845 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:18.556940 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.556991 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.557043 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:20:18.562788 32670 raft_consensus.cc:1261] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Refusing update from remote peer 8d74cec7d93f4f6e8c5a0da54b3266e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:18.562795 32389 raft_consensus.cc:1261] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Refusing update from remote peer 8d74cec7d93f4f6e8c5a0da54b3266e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:18.563442 343 consensus_queue.cc:1035] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:18.563663 356 consensus_queue.cc:1035] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:18.573858 361 mvcc.cc:205] Tried to move back new op lower bound from 7317450008826220544 to 7317450008715313152. Current Snapshot: MvccSnapshot[applied={T|T < 7317450008826220544 or (T in {7317450008826220544})}]
I20260811 22:20:18.585072 31859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:18.585294 31859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:18.588017 31859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: Sending DeleteTablet for 3 replicas of tablet aaf64f4eafc949099ab4d8028419ac28
I20260811 22:20:18.588939 32630 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf64f4eafc949099ab4d8028419ac28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:18 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38000
I20260811 22:20:18.588935 32100 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf64f4eafc949099ab4d8028419ac28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:18 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:55564
I20260811 22:20:18.588981 32350 tablet_service.cc:1514] Processing DeleteTablet for tablet aaf64f4eafc949099ab4d8028419ac28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:18 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:34360
I20260811 22:20:18.589491 380 tablet_replica.cc:330] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: stopping tablet replica
I20260811 22:20:18.589666 383 tablet_replica.cc:330] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: stopping tablet replica
I20260811 22:20:18.589952 380 raft_consensus.cc:2229] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:18.589951 383 raft_consensus.cc:2229] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:18.590400 383 raft_consensus.cc:2258] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:18.590394 380 raft_consensus.cc:2258] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:18.590375 31859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 22:20:18.590765 385 tablet_replica.cc:330] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: stopping tablet replica
W20260811 22:20:18.590819 31859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:18.592274 380 ts_tablet_manager.cc:1905] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:18.592274 383 ts_tablet_manager.cc:1905] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:18.595150 380 ts_tablet_manager.cc:1918] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:18.595152 383 ts_tablet_manager.cc:1918] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:18.595247 383 log.cc:1196] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/aaf64f4eafc949099ab4d8028419ac28
I20260811 22:20:18.595366 380 log.cc:1196] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/aaf64f4eafc949099ab4d8028419ac28
I20260811 22:20:18.595580 383 ts_tablet_manager.cc:1939] T aaf64f4eafc949099ab4d8028419ac28 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Deleting consensus metadata
I20260811 22:20:18.595798 380 ts_tablet_manager.cc:1939] T aaf64f4eafc949099ab4d8028419ac28 P 057c5b6205ed419fa0fb4db6118f0802: Deleting consensus metadata
I20260811 22:20:18.596625 32100 tablet_service.cc:1467] Processing CreateTablet for tablet cf292998eabb49f9838622c5da491337 (DEFAULT_TABLE table=test-table [id=9a19390697564f5bb6337a67e62c7a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:18.596818 385 raft_consensus.cc:2229] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:18.596887 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf292998eabb49f9838622c5da491337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:18.597159 385 raft_consensus.cc:2258] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:18.597754 32350 tablet_service.cc:1467] Processing CreateTablet for tablet cf292998eabb49f9838622c5da491337 (DEFAULT_TABLE table=test-table [id=9a19390697564f5bb6337a67e62c7a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:18.598038 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf292998eabb49f9838622c5da491337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:18.598675 32630 tablet_service.cc:1467] Processing CreateTablet for tablet cf292998eabb49f9838622c5da491337 (DEFAULT_TABLE table=test-table [id=9a19390697564f5bb6337a67e62c7a3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:18.598975 32630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf292998eabb49f9838622c5da491337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:18.599149 333 tablet_bootstrap.cc:492] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Bootstrap starting.
I20260811 22:20:18.600099 333 tablet_bootstrap.cc:654] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:18.600247 334 tablet_bootstrap.cc:492] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Bootstrap starting.
I20260811 22:20:18.600833 31826 catalog_manager.cc:4841] TS 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387): tablet aaf64f4eafc949099ab4d8028419ac28 (table test-table [id=50e016a589a34cae85278fad133b3f17]) successfully deleted
I20260811 22:20:18.601107 334 tablet_bootstrap.cc:654] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:18.601094 31826 catalog_manager.cc:4841] TS 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437): tablet aaf64f4eafc949099ab4d8028419ac28 (table test-table [id=50e016a589a34cae85278fad133b3f17]) successfully deleted
I20260811 22:20:18.601231 336 tablet_bootstrap.cc:492] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Bootstrap starting.
I20260811 22:20:18.602159 336 tablet_bootstrap.cc:654] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:18.602171 333 tablet_bootstrap.cc:492] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: No bootstrap required, opened a new log
I20260811 22:20:18.602250 333 ts_tablet_manager.cc:1397] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:18.602336 334 tablet_bootstrap.cc:492] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: No bootstrap required, opened a new log
I20260811 22:20:18.602407 334 ts_tablet_manager.cc:1397] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:18.602743 333 raft_consensus.cc:348] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.602806 334 raft_consensus.cc:348] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.602865 333 raft_consensus.cc:374] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:18.602898 334 raft_consensus.cc:374] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:18.602897 333 raft_consensus.cc:729] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d74cec7d93f4f6e8c5a0da54b3266e7, State: Initialized, Role: FOLLOWER
I20260811 22:20:18.602932 334 raft_consensus.cc:729] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057c5b6205ed419fa0fb4db6118f0802, State: Initialized, Role: FOLLOWER
I20260811 22:20:18.603018 333 consensus_queue.cc:260] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.603047 334 consensus_queue.cc:260] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.603279 334 ts_tablet_manager.cc:1428] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:18.603281 333 ts_tablet_manager.cc:1428] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:18.603878 385 ts_tablet_manager.cc:1905] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:18.605357 336 tablet_bootstrap.cc:492] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: No bootstrap required, opened a new log
I20260811 22:20:18.605450 336 ts_tablet_manager.cc:1397] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 22:20:18.605885 336 raft_consensus.cc:348] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.605994 336 raft_consensus.cc:374] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:18.606030 336 raft_consensus.cc:729] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f95f5b1a99d46d5b1026a2aef01315b, State: Initialized, Role: FOLLOWER
I20260811 22:20:18.606148 336 consensus_queue.cc:260] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.606437 336 ts_tablet_manager.cc:1428] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:18.606813 385 ts_tablet_manager.cc:1918] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:18.606950 385 log.cc:1196] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/aaf64f4eafc949099ab4d8028419ac28
I20260811 22:20:18.607290 385 ts_tablet_manager.cc:1939] T aaf64f4eafc949099ab4d8028419ac28 P 9f95f5b1a99d46d5b1026a2aef01315b: Deleting consensus metadata
I20260811 22:20:18.608698 31826 catalog_manager.cc:4841] TS 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67:34605): tablet aaf64f4eafc949099ab4d8028419ac28 (table test-table [id=50e016a589a34cae85278fad133b3f17]) successfully deleted
W20260811 22:20:18.617808 32767 tablet.cc:2369] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:18.658108 32491 tablet.cc:2369] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:18.680641 32220 tablet.cc:2369] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:18.952732 350 raft_consensus.cc:482] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:18.952907 350 raft_consensus.cc:504] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.954041 350 leader_election.cc:288] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387), 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:18.958163 331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.954877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:53277 (local address 127.29.3.66:38437)
0811 22:20:18.954998 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.955002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.955015 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:18.955134 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.955137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.955182 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.955275 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.955562 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.956043 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.957232 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.957617 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.957699 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.957769 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.957952 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.958030 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:18.958189 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.954370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:36387 (local address 127.29.3.67:37949)
0811 22:20:18.954645 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.954660 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.954817 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.955223 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.955226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.955245 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.955373 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.955381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.956157 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.956160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.957113 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.957124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.957774 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.957778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.957874 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.957893 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.957966 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.958030 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":68,"thread_start_us":84,"threads_started":1}
I20260811 22:20:18.958182 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.954511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:37949 (local address 127.29.3.65:36387)
0811 22:20:18.954647 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:18.954651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:18.954778 (+ 127us) server_negotiation.cc:407] Connection header received
0811 22:20:18.954965 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:18.954970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:18.955020 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:18.955072 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:18.955494 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.956040 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.957240 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.957649 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.957749 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.957816 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:18.958000 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 22:20:18.958081 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:18.958194 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:18.954739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38437 (local address 127.29.3.67:53277)
0811 22:20:18.954946 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:18.954958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:18.955042 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:18.955294 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:18.955296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:18.955312 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:18.955426 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.955431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.956150 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.956152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.957099 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:18.957109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:18.957734 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:18.957738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:18.957830 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:18.957854 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:20:18.957920 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 22:20:18.957999 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":70,"thread_start_us":60,"threads_started":1}
I20260811 22:20:18.958849 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf292998eabb49f9838622c5da491337" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057c5b6205ed419fa0fb4db6118f0802" is_pre_election: true
I20260811 22:20:18.958916 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf292998eabb49f9838622c5da491337" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" is_pre_election: true
I20260811 22:20:18.959012 32389 raft_consensus.cc:2454] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 0.
I20260811 22:20:18.959156 32132 raft_consensus.cc:2454] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 0.
I20260811 22:20:18.959383 32540 leader_election.cc:302] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057c5b6205ed419fa0fb4db6118f0802, 9f95f5b1a99d46d5b1026a2aef01315b; no voters:
I20260811 22:20:18.959648 350 raft_consensus.cc:2781] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:18.959705 350 raft_consensus.cc:482] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:18.959770 350 raft_consensus.cc:3037] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:18.960824 350 raft_consensus.cc:504] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.961158 350 leader_election.cc:288] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 election: Requested vote from peers 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387), 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:18.961508 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf292998eabb49f9838622c5da491337" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7"
I20260811 22:20:18.961582 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf292998eabb49f9838622c5da491337" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057c5b6205ed419fa0fb4db6118f0802"
I20260811 22:20:18.961627 32132 raft_consensus.cc:3037] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:18.961702 32389 raft_consensus.cc:3037] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:18.962638 32389 raft_consensus.cc:2454] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 1.
I20260811 22:20:18.962649 32132 raft_consensus.cc:2454] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 1.
I20260811 22:20:18.962895 32540 leader_election.cc:302] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d74cec7d93f4f6e8c5a0da54b3266e7, 9f95f5b1a99d46d5b1026a2aef01315b; no voters:
I20260811 22:20:18.963052 350 raft_consensus.cc:2781] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:18.963222 350 raft_consensus.cc:686] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 LEADER]: Becoming Leader. State: Replica: 9f95f5b1a99d46d5b1026a2aef01315b, State: Running, Role: LEADER
I20260811 22:20:18.963472 350 consensus_queue.cc:237] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:18.965785 31856 catalog_manager.cc:5495] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67). New cstate: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } health_report { overall_health: HEALTHY } } }
Aug 11 22:20:19 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486817, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:19.080277 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.072159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42539 (local address 127.0.0.1:45690)
0811 22:20:19.072462 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:19.072477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:19.072594 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:19.072840 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:19.072842 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:19.073253 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:19.073360 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:19.073366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.074196 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.074198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:19.074857 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:19.074867 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.075353 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.075356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:19.075417 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.076018 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:19.076030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:19.077494 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:19.078975 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:19.078985 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:19.078990 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:19.079225 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:19.079433 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:19.079435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:19.079437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:19.079505 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:19.079816 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:19.079821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:19.079924 (+ 103us) client_negotiation.cc:769] Sending connection context
0811 22:20:19.080010 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 22:20:19.080121 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":101,"threads_started":1}
I20260811 22:20:19.080505 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.072214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.29.3.126:42539)
0811 22:20:19.072563 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:19.072569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:19.072584 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:19.072676 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:19.072678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:19.072736 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:19.072813 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:19.073453 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.074097 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.074955 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.075255 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.075335 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.077622 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:19.077625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:19.077626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:19.077648 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:19.078881 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:19.079302 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:19.079305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:19.079306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:19.079376 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:19.079563 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:19.079565 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:19.079566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:19.079743 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:19.079815 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:19.080245 (+ 430us) server_negotiation.cc:299] Negotiation successful
0811 22:20:19.080349 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260811 22:20:19.084368 31856 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:45690:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 22:20:19.085191 31856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:19.089816 32630 tablet_service.cc:1467] Processing CreateTablet for tablet c722e166ec1b43d0907ae4dcc74ab972 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=712d743e1e0b4652804d63c6d7d56ccd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:19.089769 32100 tablet_service.cc:1467] Processing CreateTablet for tablet c722e166ec1b43d0907ae4dcc74ab972 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=712d743e1e0b4652804d63c6d7d56ccd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:19.089792 32350 tablet_service.cc:1467] Processing CreateTablet for tablet c722e166ec1b43d0907ae4dcc74ab972 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=712d743e1e0b4652804d63c6d7d56ccd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:19.090194 32630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c722e166ec1b43d0907ae4dcc74ab972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:19.090193 32350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c722e166ec1b43d0907ae4dcc74ab972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:19.090195 32100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c722e166ec1b43d0907ae4dcc74ab972. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:19.092710 336 tablet_bootstrap.cc:492] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: Bootstrap starting.
I20260811 22:20:19.092898 333 tablet_bootstrap.cc:492] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Bootstrap starting.
I20260811 22:20:19.093531 336 tablet_bootstrap.cc:654] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:19.093551 333 tablet_bootstrap.cc:654] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:19.094694 333 tablet_bootstrap.cc:492] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: No bootstrap required, opened a new log
I20260811 22:20:19.094784 336 tablet_bootstrap.cc:492] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: No bootstrap required, opened a new log
I20260811 22:20:19.094792 333 ts_tablet_manager.cc:1397] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 22:20:19.094878 336 ts_tablet_manager.cc:1397] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:19.095196 333 raft_consensus.cc:348] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.095299 333 raft_consensus.cc:374] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:19.095268 336 raft_consensus.cc:348] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.095330 333 raft_consensus.cc:729] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d74cec7d93f4f6e8c5a0da54b3266e7, State: Initialized, Role: FOLLOWER
I20260811 22:20:19.095371 336 raft_consensus.cc:374] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:19.095407 336 raft_consensus.cc:729] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f95f5b1a99d46d5b1026a2aef01315b, State: Initialized, Role: FOLLOWER
I20260811 22:20:19.095454 333 consensus_queue.cc:260] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.095521 336 consensus_queue.cc:260] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.095728 333 ts_tablet_manager.cc:1428] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:19.095781 336 ts_tablet_manager.cc:1428] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:19.095745 350 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.095872 350 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.095945 334 tablet_bootstrap.cc:492] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: Bootstrap starting.
I20260811 22:20:19.095743 372 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.096274 372 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.096930 334 tablet_bootstrap.cc:654] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:19.098145 334 tablet_bootstrap.cc:492] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: No bootstrap required, opened a new log
I20260811 22:20:19.098222 334 ts_tablet_manager.cc:1397] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:19.098623 334 raft_consensus.cc:348] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.098727 334 raft_consensus.cc:374] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:19.098748 334 raft_consensus.cc:729] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057c5b6205ed419fa0fb4db6118f0802, State: Initialized, Role: FOLLOWER
I20260811 22:20:19.098886 334 consensus_queue.cc:260] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.099100 347 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.099253 347 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.099118 334 ts_tablet_manager.cc:1428] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:19.124094 350 raft_consensus.cc:482] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:19.124239 350 raft_consensus.cc:504] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.124675 350 leader_election.cc:288] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387), 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:19.125140 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722e166ec1b43d0907ae4dcc74ab972" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" is_pre_election: true
I20260811 22:20:19.125144 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722e166ec1b43d0907ae4dcc74ab972" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057c5b6205ed419fa0fb4db6118f0802" is_pre_election: true
I20260811 22:20:19.125304 32389 raft_consensus.cc:2454] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 0.
I20260811 22:20:19.125306 32132 raft_consensus.cc:2454] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 0.
I20260811 22:20:19.125583 32540 leader_election.cc:302] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d74cec7d93f4f6e8c5a0da54b3266e7, 9f95f5b1a99d46d5b1026a2aef01315b; no voters:
I20260811 22:20:19.125771 350 raft_consensus.cc:2781] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:19.125828 350 raft_consensus.cc:482] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:19.125872 350 raft_consensus.cc:3037] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:19.126940 350 raft_consensus.cc:504] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.127247 350 leader_election.cc:288] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 election: Requested vote from peers 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387), 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437)
I20260811 22:20:19.127604 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722e166ec1b43d0907ae4dcc74ab972" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7"
I20260811 22:20:19.127641 32389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c722e166ec1b43d0907ae4dcc74ab972" candidate_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057c5b6205ed419fa0fb4db6118f0802"
I20260811 22:20:19.127751 32132 raft_consensus.cc:3037] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:19.127804 32389 raft_consensus.cc:3037] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:19.128682 32132 raft_consensus.cc:2454] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 1.
I20260811 22:20:19.128692 32389 raft_consensus.cc:2454] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f95f5b1a99d46d5b1026a2aef01315b in term 1.
I20260811 22:20:19.128952 32540 leader_election.cc:302] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057c5b6205ed419fa0fb4db6118f0802, 9f95f5b1a99d46d5b1026a2aef01315b; no voters:
I20260811 22:20:19.129102 350 raft_consensus.cc:2781] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:19.129187 350 raft_consensus.cc:686] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 LEADER]: Becoming Leader. State: Replica: 9f95f5b1a99d46d5b1026a2aef01315b, State: Running, Role: LEADER
I20260811 22:20:19.129328 350 consensus_queue.cc:237] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } }
I20260811 22:20:19.129997 395 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: TxnStatusTablet state changed. Reason: New leader 9f95f5b1a99d46d5b1026a2aef01315b. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.130089 395 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:19.130399 424 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 22:20:19.130620 424 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 22:20:19.131207 31861 catalog_manager.cc:5495] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67). New cstate: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } health_report { overall_health: HEALTHY } } }
I20260811 22:20:19.154270 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.150106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.29.3.67:34605)
0811 22:20:19.150388 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:19.150393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:19.150407 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:20:19.150475 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:19.150477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:19.150530 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:19.150649 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:19.150924 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.151454 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.152301 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.152522 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.152613 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.153384 (+ 771us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:19.153840 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:19.153947 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:19.154040 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 22:20:19.154103 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260811 22:20:19.154253 406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.150002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:34605 (local address 127.0.0.1:38004)
0811 22:20:19.150163 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:19.150180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:19.150319 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:19.150683 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:19.150687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:19.150697 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:19.150801 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:19.150807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.151567 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.151571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:19.152183 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:19.152192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.152635 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.152639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:19.152726 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.152771 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:19.153939 (+ 1168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:19.153958 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:20:19.154022 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:20:19.154126 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 22:20:19.157630 32389 raft_consensus.cc:1261] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Refusing update from remote peer 9f95f5b1a99d46d5b1026a2aef01315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:19.158012 32132 raft_consensus.cc:1261] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Refusing update from remote peer 9f95f5b1a99d46d5b1026a2aef01315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:19.158349 395 consensus_queue.cc:1035] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:19.158591 350 consensus_queue.cc:1035] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:19.159996 347 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: TxnStatusTablet state changed. Reason: New leader 9f95f5b1a99d46d5b1026a2aef01315b. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.160107 347 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.160511 372 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: TxnStatusTablet state changed. Reason: New leader 9f95f5b1a99d46d5b1026a2aef01315b. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.160631 372 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.163712 350 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.163833 350 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:19.164335 395 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.164443 395 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 9f95f5b1a99d46d5b1026a2aef01315b: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:19.165645 372 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.165760 372 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 8d74cec7d93f4f6e8c5a0da54b3266e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.166953 347 tablet_replica.cc:439] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 } } peers { permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 } } peers { permanent_uuid: "9f95f5b1a99d46d5b1026a2aef01315b" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34605 } } }
I20260811 22:20:19.167078 347 tablet_replica.cc:442] T c722e166ec1b43d0907ae4dcc74ab972 P 057c5b6205ed419fa0fb4db6118f0802: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:19.167182 433 mvcc.cc:205] Tried to move back new op lower bound from 7317450011264884736 to 7317450011155132416. Current Snapshot: MvccSnapshot[applied={T|T < 7317450011264884736}]
I20260811 22:20:19.167702 432 mvcc.cc:205] Tried to move back new op lower bound from 7317450011264884736 to 7317450011155132416. Current Snapshot: MvccSnapshot[applied={T|T < 7317450011264884736}]
I20260811 22:20:19.176913 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.173462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.29.3.67:34605)
0811 22:20:19.173684 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:19.173689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:19.173719 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:20:19.173861 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:19.173865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:19.173923 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:19.173981 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:19.174241 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.174993 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.175881 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.176157 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.176265 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.176509 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:19.176587 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:19.176642 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:19.176741 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 22:20:19.176788 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260811 22:20:19.187747 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.184267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:40433 (local address 127.29.3.67:34605)
0811 22:20:19.184377 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:19.184381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:19.184618 (+ 237us) server_negotiation.cc:407] Connection header received
0811 22:20:19.184743 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:19.184745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:19.184785 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:19.184846 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:19.185090 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.185566 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.186789 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.186987 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.187061 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.187347 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:19.187409 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:19.187461 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:19.187550 (+ 89us) server_negotiation.cc:299] Negotiation successful
0811 22:20:19.187606 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260811 22:20:19.187868 449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.184163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:34605 (local address 127.29.3.67:40433)
0811 22:20:19.184538 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:19.184553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:19.184653 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:19.184867 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:19.184870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:19.184876 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:19.185002 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:19.185008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.185661 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.185664 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:19.186685 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:19.186696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.187083 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.187086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:19.187175 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.187229 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:19.187478 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:19.187484 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:20:19.187544 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 22:20:19.187648 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":18,"thread_start_us":109,"threads_started":1}
I20260811 22:20:19.194592 32389 raft_consensus.cc:1261] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Refusing update from remote peer 9f95f5b1a99d46d5b1026a2aef01315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:19.194618 32132 raft_consensus.cc:1261] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Refusing update from remote peer 9f95f5b1a99d46d5b1026a2aef01315b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:19.194973 350 consensus_queue.cc:1035] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d74cec7d93f4f6e8c5a0da54b3266e7" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:19.195134 395 consensus_queue.cc:1035] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [LEADER]: Connected to new peer: Peer: permanent_uuid: "057c5b6205ed419fa0fb4db6118f0802" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:19.235932 31861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:19.236124 31861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:19.238243 31861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1eea6086b851460a885e657bdbda7af3: Sending DeleteTablet for 3 replicas of tablet cf292998eabb49f9838622c5da491337
I20260811 22:20:19.238849 32100 tablet_service.cc:1514] Processing DeleteTablet for tablet cf292998eabb49f9838622c5da491337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:19 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:55564
I20260811 22:20:19.238916 32350 tablet_service.cc:1514] Processing DeleteTablet for tablet cf292998eabb49f9838622c5da491337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:19 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:34360
I20260811 22:20:19.239269 459 tablet_replica.cc:330] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: stopping tablet replica
I20260811 22:20:19.239286 462 tablet_replica.cc:330] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: stopping tablet replica
I20260811 22:20:19.239414 462 raft_consensus.cc:2229] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:19.239542 462 raft_consensus.cc:2258] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:19.239526 459 raft_consensus.cc:2229] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:19.239717 459 raft_consensus.cc:2258] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:19.240278 462 ts_tablet_manager.cc:1905] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:19.240398 32630 tablet_service.cc:1514] Processing DeleteTablet for tablet cf292998eabb49f9838622c5da491337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:19 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38000
I20260811 22:20:19.240589 459 ts_tablet_manager.cc:1905] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:19.241235 469 tablet_replica.cc:330] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: stopping tablet replica
I20260811 22:20:19.241361 469 raft_consensus.cc:2229] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:19.241643 469 raft_consensus.cc:2258] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:19.242691 469 ts_tablet_manager.cc:1905] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:19.243358 462 ts_tablet_manager.cc:1918] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:19.243446 462 log.cc:1196] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/cf292998eabb49f9838622c5da491337
I20260811 22:20:19.243713 462 ts_tablet_manager.cc:1939] T cf292998eabb49f9838622c5da491337 P 057c5b6205ed419fa0fb4db6118f0802: Deleting consensus metadata
I20260811 22:20:19.244186 459 ts_tablet_manager.cc:1918] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:19.244391 459 log.cc:1196] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/cf292998eabb49f9838622c5da491337
I20260811 22:20:19.244436 31826 catalog_manager.cc:4841] TS 057c5b6205ed419fa0fb4db6118f0802 (127.29.3.66:38437): tablet cf292998eabb49f9838622c5da491337 (table test-table [id=9a19390697564f5bb6337a67e62c7a3d]) successfully deleted
I20260811 22:20:19.244760 459 ts_tablet_manager.cc:1939] T cf292998eabb49f9838622c5da491337 P 8d74cec7d93f4f6e8c5a0da54b3266e7: Deleting consensus metadata
I20260811 22:20:19.245491 31826 catalog_manager.cc:4841] TS 8d74cec7d93f4f6e8c5a0da54b3266e7 (127.29.3.65:36387): tablet cf292998eabb49f9838622c5da491337 (table test-table [id=9a19390697564f5bb6337a67e62c7a3d]) successfully deleted
I20260811 22:20:19.245700 469 ts_tablet_manager.cc:1918] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:19.245771 469 log.cc:1196] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/cf292998eabb49f9838622c5da491337
I20260811 22:20:19.245985 469 ts_tablet_manager.cc:1939] T cf292998eabb49f9838622c5da491337 P 9f95f5b1a99d46d5b1026a2aef01315b: Deleting consensus metadata
I20260811 22:20:19.246743 31826 catalog_manager.cc:4841] TS 9f95f5b1a99d46d5b1026a2aef01315b (127.29.3.67:34605): tablet cf292998eabb49f9838622c5da491337 (table test-table [id=9a19390697564f5bb6337a67e62c7a3d]) successfully deleted
Aug 11 22:20:19 dist-test-slave-4tdj krb5kdc[31766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486818, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:19.256414 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.240973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.29.3.65:36387)
0811 22:20:19.241088 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:19.241093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:19.246382 (+ 5289us) server_negotiation.cc:407] Connection header received
0811 22:20:19.246528 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:19.246531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:19.246575 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:19.246640 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:19.247467 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.248258 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.249104 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.249439 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.249549 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.252696 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:19.252711 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:19.252729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:19.252758 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:19.254600 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:19.255011 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:19.255018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:19.255020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:19.255116 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:19.255347 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:19.255351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:19.255352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:19.255676 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:19.255883 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:19.256117 (+ 234us) server_negotiation.cc:299] Negotiation successful
0811 22:20:19.256245 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260811 22:20:19.257081 32210 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55576
I20260811 22:20:19.266935 411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:19.259127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.29.3.126:42539)
0811 22:20:19.259258 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:19.259261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:19.259276 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:19.259406 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:19.259409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:19.259451 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:19.259529 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:19.260350 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.260986 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.262067 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:19.262334 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:19.262421 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:19.263983 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:19.263986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:19.263988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:19.264015 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:19.265304 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:19.265767 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:19.265769 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:19.265770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:19.265844 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:19.266119 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:19.266121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:19.266122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:19.266326 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:19.266437 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:19.266679 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:20:19.266799 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260811 22:20:19.267371 31861 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45702
I20260811 22:20:19.268836 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31971
I20260811 22:20:19.290795 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 32234
I20260811 22:20:19.301285 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 32508
I20260811 22:20:19.314023 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31791
2026-08-11T22:20:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3145 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:19 dist-test-slave-4tdj krb5kdc[479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:19 dist-test-slave-4tdj krb5kdc[479](info): set up 2 sockets
Aug 11 22:20:19 dist-test-slave-4tdj krb5kdc[479](info): commencing operation
krb5kdc: starting...
W20260811 22:20:20.499825 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:20 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:20.627473 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38535
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:38535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:20.741137 509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:20.741405 509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:20.741442 509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:20.745425 509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:20.745486 509 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 22:20:20.745501 509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:20.745524 509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:20.745543 509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:20.745560 509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:20.750814 509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:38535
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:20.751844 509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:20.752820 509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:20.761122 516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:20.761173 514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:20.761139 519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:20.761327 509 server_base.cc:1034] running on GCE node
I20260811 22:20:20.762090 509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:20.763473 509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:20.764640 509 hybrid_clock.cc:648] HybridClock initialized: now 1786486820764616 us; error 38 us; skew 500 ppm
Aug 11 22:20:20 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486820, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:20.767508 509 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:20.768399 509 webserver.cc:460] Webserver started at http://127.29.3.126:33107/ using document root <none> and password file <none>
I20260811 22:20:20.768952 509 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:20.769016 509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:20.769230 509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:20.771374 509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "510781c0c3cf4275aec8eba21a2b3b92"
format_stamp: "Formatted at 2026-08-11 22:20:20 on dist-test-slave-4tdj"
server_key: "f677e4ee2494d03e6961213c9cc3ae59"
server_key_iv: "c4fa53fb0008b1fcc81a0051699723cc"
server_key_version: "encryptionkey@0"
I20260811 22:20:20.771936 509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "510781c0c3cf4275aec8eba21a2b3b92"
format_stamp: "Formatted at 2026-08-11 22:20:20 on dist-test-slave-4tdj"
server_key: "f677e4ee2494d03e6961213c9cc3ae59"
server_key_iv: "c4fa53fb0008b1fcc81a0051699723cc"
server_key_version: "encryptionkey@0"
I20260811 22:20:20.775753 509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:20:20.778154 533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:20.779444 509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:20.779620 509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "510781c0c3cf4275aec8eba21a2b3b92"
format_stamp: "Formatted at 2026-08-11 22:20:20 on dist-test-slave-4tdj"
server_key: "f677e4ee2494d03e6961213c9cc3ae59"
server_key_iv: "c4fa53fb0008b1fcc81a0051699723cc"
server_key_version: "encryptionkey@0"
I20260811 22:20:20.779722 509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:20.795291 509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:20.798538 509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:20.798727 509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:20.806277 509 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:38535
I20260811 22:20:20.806348 637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:38535 every 8 connection(s)
I20260811 22:20:20.807241 509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:20.810038 639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:20.814378 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 509
I20260811 22:20:20.814486 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:20.814726 29709 external_mini_cluster.cc:1428] Setting key dc5dcec40ebefa14434b0b16b6e98473
I20260811 22:20:20.815945 639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Bootstrap starting.
I20260811 22:20:20.818643 639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:20.819523 639 log.cc:824] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:20.821547 639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: No bootstrap required, opened a new log
Aug 11 22:20:20 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:20.825479 639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510781c0c3cf4275aec8eba21a2b3b92" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38535 } }
I20260811 22:20:20.825755 639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:20.825825 639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 510781c0c3cf4275aec8eba21a2b3b92, State: Initialized, Role: FOLLOWER
I20260811 22:20:20.826351 639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510781c0c3cf4275aec8eba21a2b3b92" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38535 } }
I20260811 22:20:20.826494 639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:20.826556 639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:20.826650 639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:20.827740 639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510781c0c3cf4275aec8eba21a2b3b92" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38535 } }
I20260811 22:20:20.828104 639 leader_election.cc:302] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 510781c0c3cf4275aec8eba21a2b3b92; no voters:
I20260811 22:20:20.828461 639 leader_election.cc:288] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:20.828552 650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:20.828794 650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [term 1 LEADER]: Becoming Leader. State: Replica: 510781c0c3cf4275aec8eba21a2b3b92, State: Running, Role: LEADER
I20260811 22:20:20.829159 650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510781c0c3cf4275aec8eba21a2b3b92" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38535 } }
I20260811 22:20:20.829298 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:20.816133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39484 (local address 127.29.3.126:38535)
0811 22:20:20.816558 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:20.816579 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:20.816620 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:20:20.817439 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:20.817463 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:20.817873 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:20.818232 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:20.818943 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:20.820270 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:20.821104 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:20.821769 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:20.821940 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:20.824301 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:20.824308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:20.824324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:20.824356 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:20.826208 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:20.826685 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:20.826691 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:20.826697 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:20.826840 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:20.827097 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:20.827100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:20.827102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:20.827625 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:20.827833 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:20.828039 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 22:20:20.828336 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260811 22:20:20.829720 639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:20.832093 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 510781c0c3cf4275aec8eba21a2b3b92. Latest consensus state: current_term: 1 leader_uuid: "510781c0c3cf4275aec8eba21a2b3b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510781c0c3cf4275aec8eba21a2b3b92" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38535 } } }
I20260811 22:20:20.832201 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:20.832051 652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "510781c0c3cf4275aec8eba21a2b3b92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "510781c0c3cf4275aec8eba21a2b3b92" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38535 } } }
I20260811 22:20:20.832404 652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92 [sys.catalog]: This master's current role is: LEADER
W20260811 22:20:20.834674 671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:20:20.834800 671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:20:20.834920 673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:20.835624 673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:20.840767 673 catalog_manager.cc:1347] Generated new cluster ID: 6bbcc66be0ff4652aa40568c93cabc62
I20260811 22:20:20.840827 673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:20.856788 673 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:20.858109 673 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:20.868997 673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Generated new TSK 0
I20260811 22:20:20.869884 673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:20.901212 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38535
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:21.013585 688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:21.013849 688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:21.013897 688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:21.017921 688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:21.017985 688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:21.018024 688 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 22:20:21.018036 688 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:21.018076 688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:21.023844 688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:21.025163 688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:21.026341 688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:21.036577 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:21.036569 697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:21.036571 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:21.036887 688 server_base.cc:1034] running on GCE node
I20260811 22:20:21.037308 688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:21.037969 688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:21.039149 688 hybrid_clock.cc:648] HybridClock initialized: now 1786486821039126 us; error 37 us; skew 500 ppm
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:21.041980 688 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:21.042925 688 webserver.cc:460] Webserver started at http://127.29.3.65:35727/ using document root <none> and password file <none>
I20260811 22:20:21.043501 688 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:21.043573 688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:21.043785 688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:21.045855 688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "607183375abf4bc7a960879b63d0c51d"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "1cb194ef0829cd4a3ce42bc5af962eb7"
server_key_iv: "e8eb7ef0f7db9be5095e2074bb271280"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.046432 688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "607183375abf4bc7a960879b63d0c51d"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "1cb194ef0829cd4a3ce42bc5af962eb7"
server_key_iv: "e8eb7ef0f7db9be5095e2074bb271280"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.050017 688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:21.052384 711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:21.053582 688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:21.053730 688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "607183375abf4bc7a960879b63d0c51d"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "1cb194ef0829cd4a3ce42bc5af962eb7"
server_key_iv: "e8eb7ef0f7db9be5095e2074bb271280"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.053831 688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:21.084807 688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:21.088003 688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:21.088186 688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:21.089650 688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:21.089701 688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:21.089764 688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:21.089789 688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:21.100206 688 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:32815
I20260811 22:20:21.100271 908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:32815 every 8 connection(s)
I20260811 22:20:21.101186 688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:21.102939 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.090538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:37159 (local address 127.29.3.126:38535)
0811 22:20:21.090650 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.090654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.091760 (+ 1106us) server_negotiation.cc:407] Connection header received
0811 22:20:21.092757 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.092761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.092828 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.092923 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.094073 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.094909 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.096136 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.096502 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.096618 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.099289 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.099292 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.099294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.099319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.100826 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.101405 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.101408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.101409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.101485 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.101878 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.101881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.101883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.102092 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.102180 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.102695 (+ 515us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.102818 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 22:20:21.103659 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.091121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.29.3.65:37159)
0811 22:20:21.091603 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.091646 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.092469 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.093091 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.093100 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.093426 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.093927 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.093946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.095064 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.095068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.096028 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.096039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.096679 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.096684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.096868 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.097419 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.097435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.099118 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.100937 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.100943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.100956 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.101309 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.101577 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.101580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.101582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.101760 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.102192 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.102197 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.102481 (+ 284us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.102638 (+ 157us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.102883 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260811 22:20:21.104859 911 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38535
I20260811 22:20:21.105005 911 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:21.105485 911 heartbeater.cc:511] Master 127.29.3.126:38535 requested a full tablet report, sending...
I20260811 22:20:21.106961 564 ts_manager.cc:194] Registered new tserver with Master: 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815)
I20260811 22:20:21.108392 564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:37159
I20260811 22:20:21.108707 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 688
I20260811 22:20:21.108834 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:21.109126 29709 external_mini_cluster.cc:1428] Setting key 369bbec52203e76016ce01ef85bc049d
I20260811 22:20:21.113907 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.107028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54003 (local address 127.29.3.126:38535)
0811 22:20:21.107140 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.107144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.107307 (+ 163us) server_negotiation.cc:407] Connection header received
0811 22:20:21.107382 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.107385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.107423 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.107500 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.107866 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.108586 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.109603 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.109877 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.109968 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.111055 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.111058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.111060 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.111081 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.112341 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.112800 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.112803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.112805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.112885 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.113131 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.113134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.113136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.113312 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.113399 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.113641 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.113760 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260811 22:20:21.113917 920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.106944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.29.3.65:54003)
0811 22:20:21.107233 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.107245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.107322 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.107511 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.107513 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.107672 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.107767 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.107772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.108717 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.108720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.109492 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.109503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.109983 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.109986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.110078 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.110548 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.110559 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.110929 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.112451 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.112454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.112456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.112699 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.112968 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.112971 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.112973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.113051 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.113402 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.113405 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.113493 (+ 88us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.113578 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.113716 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:21.137251 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38535
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:21.248637 923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:21.248891 923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:21.248937 923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:21.252846 923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:21.252897 923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:21.252935 923 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 22:20:21.252954 923 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:21.252983 923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:21.258173 923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:21.259227 923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:21.260205 923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:21.270069 929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:21.270113 930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:21.270231 935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:21.270113 923 server_base.cc:1034] running on GCE node
I20260811 22:20:21.270742 923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:21.271363 923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:21.272526 923 hybrid_clock.cc:648] HybridClock initialized: now 1786486821272504 us; error 33 us; skew 500 ppm
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:21.275261 923 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:21.276129 923 webserver.cc:460] Webserver started at http://127.29.3.66:45551/ using document root <none> and password file <none>
I20260811 22:20:21.276674 923 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:21.276750 923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:21.276944 923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:21.279049 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "75e4898d57c13ee7314b895c7813adca"
server_key_iv: "8adb6ed37053d2ab25ffc5c146e24a11"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.279618 923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "75e4898d57c13ee7314b895c7813adca"
server_key_iv: "8adb6ed37053d2ab25ffc5c146e24a11"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.283310 923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:20:21.285742 952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:21.286917 923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:21.287024 923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "75e4898d57c13ee7314b895c7813adca"
server_key_iv: "8adb6ed37053d2ab25ffc5c146e24a11"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.287115 923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:21.299952 923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:21.303012 923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:21.303184 923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:21.304679 923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:21.304737 923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:21.304792 923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:21.304821 923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:21.314705 923 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:43553
I20260811 22:20:21.314810 1164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:43553 every 8 connection(s)
I20260811 22:20:21.315675 923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:21.318918 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.305474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:33919 (local address 127.29.3.126:38535)
0811 22:20:21.305621 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.305627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.306552 (+ 925us) server_negotiation.cc:407] Connection header received
0811 22:20:21.307617 (+ 1065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.307621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.307681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.307742 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.309409 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.310219 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.311944 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.312168 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.312253 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.315275 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.315278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.315281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.315308 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.316829 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.317368 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.317371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.317373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.317465 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.317782 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.317785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.317787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.317982 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.318071 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.318612 (+ 541us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.318753 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:21.319664 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.305875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.29.3.66:33919)
0811 22:20:21.306396 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.306446 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.307365 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.308172 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.308181 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.308592 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.309265 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.309287 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.310381 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.310385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.311819 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.311831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.312330 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.312335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.312542 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.313235 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.313258 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.315094 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.316936 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.316943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.316959 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.317267 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.317550 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.317553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.317555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.317691 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.318083 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.318089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.318381 (+ 292us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.318553 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.318871 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260811 22:20:21.320832 1165 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38535
I20260811 22:20:21.320977 1165 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:21.321547 1165 heartbeater.cc:511] Master 127.29.3.126:38535 requested a full tablet report, sending...
I20260811 22:20:21.322770 563 ts_manager.cc:194] Registered new tserver with Master: 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:21.323559 563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:33919
I20260811 22:20:21.324026 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 923
I20260811 22:20:21.324148 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:21.324457 29709 external_mini_cluster.cc:1428] Setting key 5fcea3a77deb14cd1b61a376523987e0
I20260811 22:20:21.330454 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.323181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:40639 (local address 127.29.3.126:38535)
0811 22:20:21.323306 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.323311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.323423 (+ 112us) server_negotiation.cc:407] Connection header received
0811 22:20:21.323503 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.323506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.323552 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.323651 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.324121 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.324639 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.325749 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.326020 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.326119 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.327239 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.327243 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.327246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.327272 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.328688 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.329162 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.329165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.329167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.329249 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.329569 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.329573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.329575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.329817 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.329899 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.330184 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.330324 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:20:21.330516 1171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.323080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.29.3.66:40639)
0811 22:20:21.323340 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.323359 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.323438 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.323658 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.323661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.323873 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.323996 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.324004 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.324785 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.324789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.325630 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.325643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.326136 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.326139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.326216 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.326779 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.326791 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.327100 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.328810 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.328813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.328815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.329048 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.329340 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.329343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.329345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.329467 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.329916 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.329919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.330015 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.330122 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.330283 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:21.354271 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38535
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:21.468585 1178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:21.468837 1178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:21.468885 1178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:21.472985 1178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:21.473053 1178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:21.473096 1178 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 22:20:21.473124 1178 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:21.473168 1178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:21.478585 1178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:38535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:21.479714 1178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:21.480760 1178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:21.490429 1191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:21.490480 1196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:21.490444 1190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:21.490679 1178 server_base.cc:1034] running on GCE node
I20260811 22:20:21.491266 1178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:21.491986 1178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:21.493144 1178 hybrid_clock.cc:648] HybridClock initialized: now 1786486821493127 us; error 30 us; skew 500 ppm
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:21.495808 1178 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:21.496694 1178 webserver.cc:460] Webserver started at http://127.29.3.67:46431/ using document root <none> and password file <none>
I20260811 22:20:21.497258 1178 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:21.497326 1178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:21.497534 1178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:21.499645 1178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "b43e7a7b15724a8cada5707a87f3c5de"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "019c8151250fa9af4562bc00aa453a84"
server_key_iv: "955ef2cd26efcf8ddd9fc26dee77b08b"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.500204 1178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "b43e7a7b15724a8cada5707a87f3c5de"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "019c8151250fa9af4562bc00aa453a84"
server_key_iv: "955ef2cd26efcf8ddd9fc26dee77b08b"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.504006 1178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:21.506512 1209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:21.507733 1178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:21.507849 1178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "b43e7a7b15724a8cada5707a87f3c5de"
format_stamp: "Formatted at 2026-08-11 22:20:21 on dist-test-slave-4tdj"
server_key: "019c8151250fa9af4562bc00aa453a84"
server_key_iv: "955ef2cd26efcf8ddd9fc26dee77b08b"
server_key_version: "encryptionkey@0"
I20260811 22:20:21.507947 1178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:21.539577 1178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:21.542726 1178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:21.542912 1178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:21.544440 1178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:21.544492 1178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:21.544550 1178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:21.544579 1178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:21.555116 1178 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:40613
I20260811 22:20:21.555243 1424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:40613 every 8 connection(s)
I20260811 22:20:21.556030 1178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:21.558499 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.545262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43209 (local address 127.29.3.126:38535)
0811 22:20:21.545382 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.545386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.546536 (+ 1150us) server_negotiation.cc:407] Connection header received
0811 22:20:21.547597 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.547602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.547666 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.547725 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.549167 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.549874 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.551623 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.551941 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.552062 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.554880 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.554884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.554887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.554919 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.556348 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.556917 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.556920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.556922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.557015 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.557355 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.557359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.557361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.557575 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.557665 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.558164 (+ 499us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.558317 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:20:21.559260 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.545847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.29.3.67:43209)
0811 22:20:21.546385 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.546430 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.547291 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.548100 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.548106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.548500 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.549016 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.549030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.550004 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.550008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.551488 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.551501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.552130 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.552135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.552343 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.552943 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.552965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.554701 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.556453 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.556459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.556474 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.556812 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.557099 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.557102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.557104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.557264 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.557678 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.557685 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.557922 (+ 237us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.558081 (+ 159us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.558459 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":141,"threads_started":1}
I20260811 22:20:21.560390 1426 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38535
I20260811 22:20:21.560542 1426 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:21.561040 1426 heartbeater.cc:511] Master 127.29.3.126:38535 requested a full tablet report, sending...
I20260811 22:20:21.561448 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1178
I20260811 22:20:21.561558 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:21.561848 29709 external_mini_cluster.cc:1428] Setting key 2bb6ab7b0f2583856f48962a806f10ae
I20260811 22:20:21.562219 564 ts_manager.cc:194] Registered new tserver with Master: b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67:40613)
I20260811 22:20:21.562994 564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:43209
I20260811 22:20:21.564078 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:20:21.569402 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.562592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:54599 (local address 127.29.3.126:38535)
0811 22:20:21.562708 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.562712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.563007 (+ 295us) server_negotiation.cc:407] Connection header received
0811 22:20:21.563138 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.563141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.563182 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.563242 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.563835 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.564391 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.565172 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.565403 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.565490 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.566570 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.566574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.566577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.566603 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.567880 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.568268 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.568271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.568273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.568354 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.568602 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.568605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.568607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.568797 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.568886 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.569138 (+ 252us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.569246 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:20:21.569414 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.562626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.29.3.67:54599)
0811 22:20:21.562910 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.562924 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.563026 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.563374 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.563377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.563597 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.563723 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.563730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.564518 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.564521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.565063 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.565071 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.565505 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.565509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.565596 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.566060 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.566071 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.566444 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.567982 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.567986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.567989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.568179 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.568436 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.568439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.568441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.568521 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.568887 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.568890 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.568983 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.569069 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.569226 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:21.593618 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.584823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.29.3.126:38535)
0811 22:20:21.584951 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.584955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.584968 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:21.585015 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.585017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.585068 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.585146 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.586059 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.586641 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.587475 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.587699 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.587792 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.590391 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.590394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.590396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.590426 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.591744 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.592216 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.592220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.592221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.592316 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.592683 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.592686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.592687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.592900 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.593022 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.593313 (+ 291us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.593446 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 22:20:21.596674 563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:21.599884 563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:21.614759 1465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.608357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:43553 (local address 127.0.0.1:39132)
0811 22:20:21.609203 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.609256 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.609393 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.609912 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.609918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.609942 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:21.610070 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.610077 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.611628 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.611632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.612837 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.612849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.614119 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.614122 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.614278 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.614344 (+ 66us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.614450 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.614603 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":56,"thread_start_us":86,"threads_started":1}
I20260811 22:20:21.614769 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.608515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.29.3.66:43553)
0811 22:20:21.609241 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.609253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.609467 (+ 214us) server_negotiation.cc:407] Connection header received
0811 22:20:21.609545 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.609550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.609763 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.609911 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:21.610193 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.611388 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.613189 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.613995 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.614276 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.614468 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.614521 (+ 53us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.614610 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":109,"threads_started":1}
I20260811 22:20:21.614850 1470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.609041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53652 (local address 127.29.3.67:40613)
0811 22:20:21.609418 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.609426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.610264 (+ 838us) server_negotiation.cc:407] Connection header received
0811 22:20:21.610382 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.610386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.610574 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.610713 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:21.610881 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.611752 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.613159 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.613734 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.613835 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.614017 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.614595 (+ 578us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.614697 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260811 22:20:21.615674 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.608555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:32815 (local address 127.0.0.1:44890)
0811 22:20:21.609802 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.609815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.609912 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.610433 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.610436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.610451 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:21.610556 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.610562 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.611845 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.611848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.613694 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.613706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.615314 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.615316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.615383 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.615401 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.615473 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.615543 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"mutex_wait_us":60,"thread_start_us":118,"threads_started":1}
I20260811 22:20:21.615733 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.609153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.29.3.65:32815)
0811 22:20:21.609675 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.609681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.610005 (+ 324us) server_negotiation.cc:407] Connection header received
0811 22:20:21.610088 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.610093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.610287 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.610460 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:21.610689 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.611738 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.613829 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.614641 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.614749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.615255 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.615523 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.615618 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":103,"threads_started":1}
I20260811 22:20:21.617920 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 1f914ea0b075494da3f650cd3486e371 (DEFAULT_TABLE table=test-table [id=082175ed4c2345828836bd8ba6c97290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:21.618685 795 tablet_service.cc:1467] Processing CreateTablet for tablet 1f914ea0b075494da3f650cd3486e371 (DEFAULT_TABLE table=test-table [id=082175ed4c2345828836bd8ba6c97290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:21.619185 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f914ea0b075494da3f650cd3486e371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:21.619786 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f914ea0b075494da3f650cd3486e371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:21.622511 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.608670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40613 (local address 127.0.0.1:53652)
0811 22:20:21.610152 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.610166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.610254 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.610687 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.610690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.610705 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:21.610793 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.610799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.611865 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.611867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.613022 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.613033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.614084 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.614088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.614201 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.614339 (+ 138us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.614453 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.622397 (+ 7944us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"mutex_wait_us":515,"thread_start_us":71,"threads_started":1}
I20260811 22:20:21.625173 1306 tablet_service.cc:1467] Processing CreateTablet for tablet 1f914ea0b075494da3f650cd3486e371 (DEFAULT_TABLE table=test-table [id=082175ed4c2345828836bd8ba6c97290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:21.626123 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f914ea0b075494da3f650cd3486e371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:21.627130 1474 tablet_bootstrap.cc:492] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Bootstrap starting.
I20260811 22:20:21.627552 1473 tablet_bootstrap.cc:492] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Bootstrap starting.
I20260811 22:20:21.629894 1474 tablet_bootstrap.cc:654] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:21.630074 1473 tablet_bootstrap.cc:654] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:21.630841 1474 log.cc:824] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:21.631083 1473 log.cc:824] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:21.632778 1473 tablet_bootstrap.cc:492] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: No bootstrap required, opened a new log
I20260811 22:20:21.632916 1474 tablet_bootstrap.cc:492] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: No bootstrap required, opened a new log
I20260811 22:20:21.632973 1473 ts_tablet_manager.cc:1397] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:20:21.633121 1474 ts_tablet_manager.cc:1397] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260811 22:20:21.633903 1483 tablet_bootstrap.cc:492] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Bootstrap starting.
I20260811 22:20:21.636202 1483 tablet_bootstrap.cc:654] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:21.636174 1474 raft_consensus.cc:348] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.636453 1474 raft_consensus.cc:374] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:21.636507 1474 raft_consensus.cc:729] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cecbe73fc5b459786221ec4ff0b6bd2, State: Initialized, Role: FOLLOWER
I20260811 22:20:21.636973 1474 consensus_queue.cc:260] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.637084 1473 raft_consensus.cc:348] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.637324 1473 raft_consensus.cc:374] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:21.637382 1473 raft_consensus.cc:729] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607183375abf4bc7a960879b63d0c51d, State: Initialized, Role: FOLLOWER
I20260811 22:20:21.637814 1165 heartbeater.cc:503] Master 127.29.3.126:38535 was elected leader, sending a full tablet report...
I20260811 22:20:21.637863 1483 log.cc:824] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:21.637815 1473 consensus_queue.cc:260] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.638072 1474 ts_tablet_manager.cc:1428] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 22:20:21.638782 1473 ts_tablet_manager.cc:1428] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:20:21.639111 911 heartbeater.cc:503] Master 127.29.3.126:38535 was elected leader, sending a full tablet report...
I20260811 22:20:21.639799 1483 tablet_bootstrap.cc:492] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: No bootstrap required, opened a new log
I20260811 22:20:21.639978 1483 ts_tablet_manager.cc:1397] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:20:21.643313 1483 raft_consensus.cc:348] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.643527 1483 raft_consensus.cc:374] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:21.643584 1483 raft_consensus.cc:729] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Initialized, Role: FOLLOWER
I20260811 22:20:21.643959 1483 consensus_queue.cc:260] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.644583 1426 heartbeater.cc:503] Master 127.29.3.126:38535 was elected leader, sending a full tablet report...
I20260811 22:20:21.644876 1483 ts_tablet_manager.cc:1428] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:20:21.755342 1488 raft_consensus.cc:482] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:21.755566 1488 raft_consensus.cc:504] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.756767 1488 leader_election.cc:288] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553), 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815)
I20260811 22:20:21.760638 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.757038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:32815 (local address 127.29.3.67:55191)
0811 22:20:21.757152 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.757179 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.757284 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.757563 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.757566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.757584 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:21.757695 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.757703 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.758629 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.758633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.759538 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.759549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.760200 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.760204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.760303 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.760330 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.760402 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.760500 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 22:20:21.760654 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.757164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:55191 (local address 127.29.3.65:32815)
0811 22:20:21.757322 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.757326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.757344 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:20:21.757408 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.757411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.757463 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.757544 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:21.757835 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.758525 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.759690 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.760093 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.760178 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.760231 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.760432 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.760507 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:20:21.761399 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f914ea0b075494da3f650cd3486e371" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d" is_pre_election: true
I20260811 22:20:21.761523 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.757041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:43553 (local address 127.29.3.67:41691)
0811 22:20:21.757628 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.757642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.757779 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.758029 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.758032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.758047 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:21.758884 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.758895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.759591 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.759595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.760419 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.760428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.761187 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.761190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.761266 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.761285 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.761343 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.761393 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":178,"thread_start_us":160,"threads_started":1}
I20260811 22:20:21.761591 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.757163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:41691 (local address 127.29.3.66:43553)
0811 22:20:21.757312 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.757317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.757713 (+ 396us) server_negotiation.cc:407] Connection header received
0811 22:20:21.757875 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.757878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.757933 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.758003 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:21.759012 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.759493 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.760516 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.761057 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.761170 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.761229 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.761369 (+ 140us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.761461 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:20:21.761770 828 raft_consensus.cc:2454] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:21.762277 1213 leader_election.cc:302] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607183375abf4bc7a960879b63d0c51d, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:21.762346 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f914ea0b075494da3f650cd3486e371" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" is_pre_election: true
I20260811 22:20:21.762604 1488 raft_consensus.cc:2781] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:21.762620 1078 raft_consensus.cc:2454] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:21.762687 1488 raft_consensus.cc:482] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:21.762756 1488 raft_consensus.cc:3037] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:21.763826 1488 raft_consensus.cc:504] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.764158 1488 leader_election.cc:288] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Requested vote from peers 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553), 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815)
I20260811 22:20:21.764460 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f914ea0b075494da3f650cd3486e371" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
I20260811 22:20:21.764495 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f914ea0b075494da3f650cd3486e371" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d"
I20260811 22:20:21.764564 1078 raft_consensus.cc:3037] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:21.764595 828 raft_consensus.cc:3037] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:21.765595 1078 raft_consensus.cc:2454] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:21.765606 828 raft_consensus.cc:2454] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:21.765908 1210 leader_election.cc:302] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 607183375abf4bc7a960879b63d0c51d, 7cecbe73fc5b459786221ec4ff0b6bd2, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:21.766093 1488 raft_consensus.cc:2781] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:21.766394 1488 raft_consensus.cc:686] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Becoming Leader. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Running, Role: LEADER
I20260811 22:20:21.766716 1488 consensus_queue.cc:237] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.769907 563 catalog_manager.cc:5495] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de reported cstate change: term changed from 0 to 1, leader changed from <none> to b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67). New cstate: current_term: 1 leader_uuid: "b43e7a7b15724a8cada5707a87f3c5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } health_report { overall_health: HEALTHY } } }
W20260811 22:20:21.807261 1427 tablet.cc:2369] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:21.816993 1166 tablet.cc:2369] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 22:20:21 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486820, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:21.836016 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.828056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38535 (local address 127.0.0.1:39506)
0811 22:20:21.828332 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:21.828344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:21.828458 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:21.828675 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:21.828678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:21.829099 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:21.829202 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.829209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.829981 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.829985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.830590 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:21.830599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.831074 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.831077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:21.831162 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.831651 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:21.831663 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:21.833026 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:21.834519 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.834529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.834536 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.834762 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.835003 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:21.835006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:21.835008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:21.835080 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:21.835440 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:21.835448 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:21.835555 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 22:20:21.835649 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 22:20:21.835821 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":98,"threads_started":1}
I20260811 22:20:21.836015 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:21.828107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39506 (local address 127.29.3.126:38535)
0811 22:20:21.828247 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:21.828252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:21.828413 (+ 161us) server_negotiation.cc:407] Connection header received
0811 22:20:21.828540 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:21.828543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:21.828581 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:21.828645 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:21.829318 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.829880 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.830692 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:21.830966 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:21.831052 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:21.833153 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:21.833156 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:21.833159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:21.833183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:21.834419 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.834848 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.834852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.834854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.834926 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:21.835156 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:21.835159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:21.835161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:21.835340 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:21.835427 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:21.835711 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 22:20:21.835879 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":38}
I20260811 22:20:21.839993 563 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39506:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 22:20:21.840734 563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:21.845419 795 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2b864cb4784085b2d79b007a3e42fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc38911da92641eb9b66bf5b478b122e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:21.845481 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2b864cb4784085b2d79b007a3e42fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc38911da92641eb9b66bf5b478b122e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:21.845642 1306 tablet_service.cc:1467] Processing CreateTablet for tablet 8e2b864cb4784085b2d79b007a3e42fe (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc38911da92641eb9b66bf5b478b122e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:21.846226 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2b864cb4784085b2d79b007a3e42fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:21.846225 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2b864cb4784085b2d79b007a3e42fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:21.846235 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e2b864cb4784085b2d79b007a3e42fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:21.848888 1473 tablet_bootstrap.cc:492] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: Bootstrap starting.
I20260811 22:20:21.849042 1483 tablet_bootstrap.cc:492] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: Bootstrap starting.
I20260811 22:20:21.849047 1474 tablet_bootstrap.cc:492] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: Bootstrap starting.
I20260811 22:20:21.849741 1473 tablet_bootstrap.cc:654] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:21.849962 1474 tablet_bootstrap.cc:654] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:21.849962 1483 tablet_bootstrap.cc:654] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:21.850986 1473 tablet_bootstrap.cc:492] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: No bootstrap required, opened a new log
I20260811 22:20:21.851085 1473 ts_tablet_manager.cc:1397] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:21.851296 1474 tablet_bootstrap.cc:492] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: No bootstrap required, opened a new log
I20260811 22:20:21.851368 1474 ts_tablet_manager.cc:1397] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 22:20:21.851406 1483 tablet_bootstrap.cc:492] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: No bootstrap required, opened a new log
I20260811 22:20:21.851464 1483 ts_tablet_manager.cc:1397] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 22:20:21.851580 1473 raft_consensus.cc:348] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.851707 1473 raft_consensus.cc:374] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:21.851740 1473 raft_consensus.cc:729] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607183375abf4bc7a960879b63d0c51d, State: Initialized, Role: FOLLOWER
I20260811 22:20:21.851796 1483 raft_consensus.cc:348] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.851889 1483 raft_consensus.cc:374] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:21.851866 1473 consensus_queue.cc:260] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.851922 1483 raft_consensus.cc:729] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Initialized, Role: FOLLOWER
I20260811 22:20:21.852038 1483 consensus_queue.cc:260] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.852121 1473 ts_tablet_manager.cc:1428] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:21.852092 1474 raft_consensus.cc:348] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.852190 1474 raft_consensus.cc:374] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:21.852138 1486 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:21.852221 1474 raft_consensus.cc:729] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cecbe73fc5b459786221ec4ff0b6bd2, State: Initialized, Role: FOLLOWER
I20260811 22:20:21.852280 1486 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:21.852300 1483 ts_tablet_manager.cc:1428] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:21.852335 1474 consensus_queue.cc:260] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:21.852519 1488 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:21.852586 1474 ts_tablet_manager.cc:1428] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:21.852632 1488 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: This TxnStatusTablet replica's current role is: FOLLOWER
W20260811 22:20:21.852985 912 tablet.cc:2369] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:21.853098 1485 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:21.853235 1485 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:22.128938 1486 raft_consensus.cc:482] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:22.129112 1486 raft_consensus.cc:504] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.130398 1486 leader_election.cc:288] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553), b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67:40613)
I20260811 22:20:22.134660 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.130781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:58701 (local address 127.29.3.66:43553)
0811 22:20:22.130947 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:22.130952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:22.131275 (+ 323us) server_negotiation.cc:407] Connection header received
0811 22:20:22.131406 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:22.131408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:22.131448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:22.131513 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:22.131835 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.132608 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.133701 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.134092 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.134172 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.134246 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:22.134446 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 22:20:22.134528 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:20:22.134701 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.130640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40613 (local address 127.29.3.65:46753)
0811 22:20:22.131087 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:22.131103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:22.131231 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:22.131503 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:22.131506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:22.131533 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:22.131657 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.131663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.132796 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.132799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.133584 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.133593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.134261 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.134265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.134389 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.134411 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:20:22.134483 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 22:20:22.134547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":2,"spinlock_wait_cycles":22528,"thread_start_us":127,"threads_started":1}
I20260811 22:20:22.134738 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.130734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:46753 (local address 127.29.3.67:40613)
0811 22:20:22.130964 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:22.130968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:22.131199 (+ 231us) server_negotiation.cc:407] Connection header received
0811 22:20:22.131342 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:22.131346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:22.131400 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:22.131482 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:22.131788 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.132609 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.133707 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.134133 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.134249 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.134345 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:22.134522 (+ 177us) server_negotiation.cc:299] Negotiation successful
0811 22:20:22.134607 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260811 22:20:22.135314 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b864cb4784085b2d79b007a3e42fe" candidate_uuid: "607183375abf4bc7a960879b63d0c51d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43e7a7b15724a8cada5707a87f3c5de" is_pre_election: true
I20260811 22:20:22.135550 1344 raft_consensus.cc:2454] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607183375abf4bc7a960879b63d0c51d in term 0.
I20260811 22:20:22.135674 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.130640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:43553 (local address 127.29.3.65:58701)
0811 22:20:22.131152 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:22.131166 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:22.131298 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:22.131544 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:22.131547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:22.131589 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:22.131691 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.131698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.132726 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.132729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.133576 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.133585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.134206 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.134211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.134327 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.134351 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:20:22.134418 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 22:20:22.134482 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":238,"spinlock_wait_cycles":896,"thread_start_us":121,"threads_started":1}
I20260811 22:20:22.136009 714 leader_election.cc:302] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607183375abf4bc7a960879b63d0c51d, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:22.136129 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b864cb4784085b2d79b007a3e42fe" candidate_uuid: "607183375abf4bc7a960879b63d0c51d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" is_pre_election: true
I20260811 22:20:22.136281 1078 raft_consensus.cc:2454] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607183375abf4bc7a960879b63d0c51d in term 0.
I20260811 22:20:22.136300 1486 raft_consensus.cc:2781] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:22.136358 1486 raft_consensus.cc:482] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:22.136399 1486 raft_consensus.cc:3037] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:22.137418 1486 raft_consensus.cc:504] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.137734 1486 leader_election.cc:288] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [CANDIDATE]: Term 1 election: Requested vote from peers 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553), b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67:40613)
I20260811 22:20:22.138056 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b864cb4784085b2d79b007a3e42fe" candidate_uuid: "607183375abf4bc7a960879b63d0c51d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
I20260811 22:20:22.138091 1344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e2b864cb4784085b2d79b007a3e42fe" candidate_uuid: "607183375abf4bc7a960879b63d0c51d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43e7a7b15724a8cada5707a87f3c5de"
I20260811 22:20:22.138182 1078 raft_consensus.cc:3037] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:22.138206 1344 raft_consensus.cc:3037] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:22.139313 1344 raft_consensus.cc:2454] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607183375abf4bc7a960879b63d0c51d in term 1.
I20260811 22:20:22.139313 1078 raft_consensus.cc:2454] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607183375abf4bc7a960879b63d0c51d in term 1.
I20260811 22:20:22.139580 714 leader_election.cc:302] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607183375abf4bc7a960879b63d0c51d, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:22.139747 1486 raft_consensus.cc:2781] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:22.139914 1486 raft_consensus.cc:686] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [term 1 LEADER]: Becoming Leader. State: Replica: 607183375abf4bc7a960879b63d0c51d, State: Running, Role: LEADER
I20260811 22:20:22.140293 1486 consensus_queue.cc:237] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.142745 1521 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: TxnStatusTablet state changed. Reason: New leader 607183375abf4bc7a960879b63d0c51d. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.142855 1521 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:22.143185 1526 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 22:20:22.143399 1526 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 22:20:22.143862 562 catalog_manager.cc:5495] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d reported cstate change: term changed from 0 to 1, leader changed from <none> to 607183375abf4bc7a960879b63d0c51d (127.29.3.65). New cstate: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:22.216421 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.212274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.29.3.65:32815)
0811 22:20:22.212389 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:22.212392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:22.212501 (+ 109us) server_negotiation.cc:407] Connection header received
0811 22:20:22.212733 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:22.212735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:22.212795 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:22.212879 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:22.213121 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.213612 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.214839 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.215245 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.215338 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.215645 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.216057 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.216143 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:22.216256 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 22:20:22.216308 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:20:22.216460 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.212185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:32815 (local address 127.0.0.1:44906)
0811 22:20:22.212442 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:22.212457 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:22.212590 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:22.212903 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:22.212906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:22.212916 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:22.213037 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.213043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.213700 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.213703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.214710 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.214722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.215358 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.215362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.215459 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.215516 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.216148 (+ 632us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.216162 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 22:20:22.216231 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 22:20:22.216318 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260811 22:20:22.221583 1344 raft_consensus.cc:1261] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Refusing update from remote peer 607183375abf4bc7a960879b63d0c51d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:22.221719 1078 raft_consensus.cc:1261] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Refusing update from remote peer 607183375abf4bc7a960879b63d0c51d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:22.222213 1521 consensus_queue.cc:1035] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:22.222666 1486 consensus_queue.cc:1035] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:22.225682 1488 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: TxnStatusTablet state changed. Reason: New leader 607183375abf4bc7a960879b63d0c51d. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.225819 1488 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:22.226837 1485 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: TxnStatusTablet state changed. Reason: New leader 607183375abf4bc7a960879b63d0c51d. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.226966 1485 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:22.230129 1486 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.230238 1486 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:22.230355 1486 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.230413 1486 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 607183375abf4bc7a960879b63d0c51d: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:22.230399 1485 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.230497 1485 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P 7cecbe73fc5b459786221ec4ff0b6bd2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:22.232141 1488 tablet_replica.cc:439] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "607183375abf4bc7a960879b63d0c51d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } } }
I20260811 22:20:22.232249 1488 tablet_replica.cc:442] T 8e2b864cb4784085b2d79b007a3e42fe P b43e7a7b15724a8cada5707a87f3c5de: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:22.232474 1532 mvcc.cc:205] Tried to move back new op lower bound from 7317450023812997120 to 7317450023489376256. Current Snapshot: MvccSnapshot[applied={T|T < 7317450023812997120}]
I20260811 22:20:22.239787 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.235944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.29.3.67:40613)
0811 22:20:22.236057 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:22.236061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:22.236316 (+ 255us) server_negotiation.cc:407] Connection header received
0811 22:20:22.236444 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:22.236446 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:22.236494 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:22.236572 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:22.236808 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.237289 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.238501 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.238704 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.238780 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.239053 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.239425 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.239526 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:22.239600 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 22:20:22.239652 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:20:22.245775 1488 consensus_queue.cc:1035] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:22.250976 1491 consensus_queue.cc:1035] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:22.252290 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.247804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:45825 (local address 127.29.3.65:32815)
0811 22:20:22.247911 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:22.247916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:22.248297 (+ 381us) server_negotiation.cc:407] Connection header received
0811 22:20:22.248494 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:22.248497 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:22.248551 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:22.248597 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:22.248988 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.249643 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.250879 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.251170 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.251249 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.251708 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.251802 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.251849 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:22.252091 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:20:22.252155 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 22:20:22.252682 1549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.247434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:32815 (local address 127.29.3.67:45825)
0811 22:20:22.248214 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:22.248230 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:22.248379 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:22.248747 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:22.248750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:22.248758 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:22.248860 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.248864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.249784 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.249787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.250756 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.250767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.251379 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.251383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.251471 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.251527 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.251912 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.251918 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:20:22.252462 (+ 544us) client_negotiation.cc:240] Negotiation successful
0811 22:20:22.252544 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":109,"threads_started":1}
I20260811 22:20:22.278630 1515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.274667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:43505 (local address 127.29.3.67:40613)
0811 22:20:22.274783 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:22.274787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:22.275337 (+ 550us) server_negotiation.cc:407] Connection header received
0811 22:20:22.275440 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:22.275442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:22.275495 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:22.275566 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:22.275937 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.276447 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.277323 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.277617 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.277677 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.278061 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.278136 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.278191 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:22.278450 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 22:20:22.278508 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 22:20:22.278972 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:22.274607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40613 (local address 127.29.3.65:43505)
0811 22:20:22.275248 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:22.275262 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:22.275361 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:22.275683 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:22.275686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:22.275695 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:22.275831 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.275836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.276545 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.276549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.277188 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:22.277198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:22.277786 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:22.277790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:22.277885 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:22.277939 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:22.278336 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:22.278342 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:20:22.278772 (+ 430us) client_negotiation.cc:240] Negotiation successful
0811 22:20:22.278852 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":103,"threads_started":1}
I20260811 22:20:22.302609 564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:22.302812 564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:22.305254 564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Sending DeleteTablet for 3 replicas of tablet 1f914ea0b075494da3f650cd3486e371
I20260811 22:20:22.305948 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f914ea0b075494da3f650cd3486e371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:22 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260811 22:20:22.305936 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f914ea0b075494da3f650cd3486e371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:22 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39132
I20260811 22:20:22.306537 1568 tablet_replica.cc:330] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: stopping tablet replica
I20260811 22:20:22.306599 564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260811 22:20:22.306717 1306 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f914ea0b075494da3f650cd3486e371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:22 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:53652
I20260811 22:20:22.306964 1568 raft_consensus.cc:2229] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Raft consensus shutting down.
W20260811 22:20:22.307004 564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:22.307190 1568 raft_consensus.cc:2258] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:22.307346 1570 tablet_replica.cc:330] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: stopping tablet replica
I20260811 22:20:22.307811 1569 tablet_replica.cc:330] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: stopping tablet replica
I20260811 22:20:22.307888 1570 raft_consensus.cc:2229] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:22.308086 1569 raft_consensus.cc:2229] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:22.308295 1569 raft_consensus.cc:2258] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:22.308593 1570 raft_consensus.cc:2258] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:22.309896 1568 ts_tablet_manager.cc:1905] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:22.311368 1570 ts_tablet_manager.cc:1905] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:22.312335 795 tablet_service.cc:1467] Processing CreateTablet for tablet 069f3e6d46554596806ca6198d0087e5 (DEFAULT_TABLE table=test-table [id=f5e77d5cb24945c49c86e9d1c007130f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:22.312563 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069f3e6d46554596806ca6198d0087e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:22.312999 1568 ts_tablet_manager.cc:1918] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:22.313078 1568 log.cc:1196] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/1f914ea0b075494da3f650cd3486e371
I20260811 22:20:22.313364 1568 ts_tablet_manager.cc:1939] T 1f914ea0b075494da3f650cd3486e371 P 607183375abf4bc7a960879b63d0c51d: Deleting consensus metadata
I20260811 22:20:22.314311 1570 ts_tablet_manager.cc:1918] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:22.314399 1570 log.cc:1196] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/1f914ea0b075494da3f650cd3486e371
I20260811 22:20:22.314693 1570 ts_tablet_manager.cc:1939] T 1f914ea0b075494da3f650cd3486e371 P b43e7a7b15724a8cada5707a87f3c5de: Deleting consensus metadata
I20260811 22:20:22.314663 1306 tablet_service.cc:1467] Processing CreateTablet for tablet 069f3e6d46554596806ca6198d0087e5 (DEFAULT_TABLE table=test-table [id=f5e77d5cb24945c49c86e9d1c007130f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:22.314889 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069f3e6d46554596806ca6198d0087e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:22.315704 540 catalog_manager.cc:4841] TS b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67:40613): tablet 1f914ea0b075494da3f650cd3486e371 (table test-table [id=082175ed4c2345828836bd8ba6c97290]) successfully deleted
I20260811 22:20:22.316949 1483 tablet_bootstrap.cc:492] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Bootstrap starting.
I20260811 22:20:22.317781 1483 tablet_bootstrap.cc:654] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:22.318815 539 catalog_manager.cc:4841] TS 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815): tablet 1f914ea0b075494da3f650cd3486e371 (table test-table [id=082175ed4c2345828836bd8ba6c97290]) successfully deleted
I20260811 22:20:22.319491 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 069f3e6d46554596806ca6198d0087e5 (DEFAULT_TABLE table=test-table [id=f5e77d5cb24945c49c86e9d1c007130f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:22.319751 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069f3e6d46554596806ca6198d0087e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:22.320188 1483 tablet_bootstrap.cc:492] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: No bootstrap required, opened a new log
I20260811 22:20:22.320268 1483 ts_tablet_manager.cc:1397] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 22:20:22.320940 1483 raft_consensus.cc:348] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.322021 1483 raft_consensus.cc:374] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:22.322088 1483 raft_consensus.cc:729] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Initialized, Role: FOLLOWER
I20260811 22:20:22.322124 1474 tablet_bootstrap.cc:492] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Bootstrap starting.
I20260811 22:20:22.322199 1483 consensus_queue.cc:260] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.322745 1473 tablet_bootstrap.cc:492] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Bootstrap starting.
I20260811 22:20:22.323077 1474 tablet_bootstrap.cc:654] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:22.323501 1473 tablet_bootstrap.cc:654] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:22.324405 1474 tablet_bootstrap.cc:492] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: No bootstrap required, opened a new log
I20260811 22:20:22.324484 1474 ts_tablet_manager.cc:1397] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260811 22:20:22.324891 1474 raft_consensus.cc:348] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.324997 1474 raft_consensus.cc:374] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:22.325027 1474 raft_consensus.cc:729] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cecbe73fc5b459786221ec4ff0b6bd2, State: Initialized, Role: FOLLOWER
I20260811 22:20:22.325052 1483 ts_tablet_manager.cc:1428] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260811 22:20:22.325139 1474 consensus_queue.cc:260] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.325404 1474 ts_tablet_manager.cc:1428] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260811 22:20:22.327370 1473 tablet_bootstrap.cc:492] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: No bootstrap required, opened a new log
I20260811 22:20:22.327457 1473 ts_tablet_manager.cc:1397] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260811 22:20:22.327831 1473 raft_consensus.cc:348] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.327945 1473 raft_consensus.cc:374] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:22.327975 1473 raft_consensus.cc:729] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607183375abf4bc7a960879b63d0c51d, State: Initialized, Role: FOLLOWER
I20260811 22:20:22.328095 1473 consensus_queue.cc:260] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.328372 1473 ts_tablet_manager.cc:1428] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:22.330742 1569 ts_tablet_manager.cc:1905] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:22.333051 1569 ts_tablet_manager.cc:1918] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:22.333150 1569 log.cc:1196] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/1f914ea0b075494da3f650cd3486e371
I20260811 22:20:22.333426 1569 ts_tablet_manager.cc:1939] T 1f914ea0b075494da3f650cd3486e371 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting consensus metadata
I20260811 22:20:22.334491 536 catalog_manager.cc:4841] TS 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553): tablet 1f914ea0b075494da3f650cd3486e371 (table test-table [id=082175ed4c2345828836bd8ba6c97290]) successfully deleted
I20260811 22:20:22.671500 1559 raft_consensus.cc:482] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:22.671655 1559 raft_consensus.cc:504] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.672128 1559 leader_election.cc:288] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815), 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:22.672690 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069f3e6d46554596806ca6198d0087e5" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" is_pre_election: true
I20260811 22:20:22.672690 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069f3e6d46554596806ca6198d0087e5" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d" is_pre_election: true
I20260811 22:20:22.672857 1078 raft_consensus.cc:2454] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:22.672857 828 raft_consensus.cc:2454] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:22.673193 1210 leader_election.cc:302] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 607183375abf4bc7a960879b63d0c51d, 7cecbe73fc5b459786221ec4ff0b6bd2, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:22.673398 1559 raft_consensus.cc:2781] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:22.673463 1559 raft_consensus.cc:482] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:22.673491 1559 raft_consensus.cc:3037] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:22.674599 1559 raft_consensus.cc:504] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.674929 1559 leader_election.cc:288] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Requested vote from peers 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815), 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:22.675304 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069f3e6d46554596806ca6198d0087e5" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d"
I20260811 22:20:22.675303 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069f3e6d46554596806ca6198d0087e5" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
I20260811 22:20:22.675438 1078 raft_consensus.cc:3037] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:22.675438 828 raft_consensus.cc:3037] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:22.676558 828 raft_consensus.cc:2454] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:22.676558 1078 raft_consensus.cc:2454] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:22.676851 1210 leader_election.cc:302] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cecbe73fc5b459786221ec4ff0b6bd2, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:22.677028 1559 raft_consensus.cc:2781] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:22.677120 1559 raft_consensus.cc:686] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Becoming Leader. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Running, Role: LEADER
I20260811 22:20:22.677259 1559 consensus_queue.cc:237] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:22.678988 563 catalog_manager.cc:5495] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de reported cstate change: term changed from 0 to 1, leader changed from <none> to b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67). New cstate: current_term: 1 leader_uuid: "b43e7a7b15724a8cada5707a87f3c5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } health_report { overall_health: HEALTHY } } }
I20260811 22:20:22.695264 1078 raft_consensus.cc:1261] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Refusing update from remote peer b43e7a7b15724a8cada5707a87f3c5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:22.695286 828 raft_consensus.cc:1261] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Refusing update from remote peer b43e7a7b15724a8cada5707a87f3c5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:22.695660 1559 consensus_queue.cc:1035] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:22.695945 1488 consensus_queue.cc:1035] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:22.700911 1531 mvcc.cc:205] Tried to move back new op lower bound from 7317450025756450816 to 7317450025687412736. Current Snapshot: MvccSnapshot[applied={T|T < 7317450025756450816}]
W20260811 22:20:22.709095 1532 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 22:20:24.258575 562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:24.258733 562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39490:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:24.261234 562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Sending DeleteTablet for 3 replicas of tablet 069f3e6d46554596806ca6198d0087e5
I20260811 22:20:24.261857 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 069f3e6d46554596806ca6198d0087e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:24 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260811 22:20:24.261854 1306 tablet_service.cc:1514] Processing DeleteTablet for tablet 069f3e6d46554596806ca6198d0087e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:24 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:53652
I20260811 22:20:24.262195 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet 069f3e6d46554596806ca6198d0087e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:24 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39132
I20260811 22:20:24.262234 1627 tablet_replica.cc:330] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: stopping tablet replica
I20260811 22:20:24.262393 1627 raft_consensus.cc:2229] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:24.262552 1629 tablet_replica.cc:330] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: stopping tablet replica
I20260811 22:20:24.262598 1627 raft_consensus.cc:2258] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:24.262682 1629 raft_consensus.cc:2229] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:24.262840 1623 tablet_replica.cc:330] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: stopping tablet replica
I20260811 22:20:24.262862 1629 raft_consensus.cc:2258] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:24.262971 1623 raft_consensus.cc:2229] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:24.263433 1629 ts_tablet_manager.cc:1905] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:24.263458 1627 ts_tablet_manager.cc:1905] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:24.266042 1629 ts_tablet_manager.cc:1918] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:20:24.266117 1629 log.cc:1196] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/069f3e6d46554596806ca6198d0087e5
I20260811 22:20:24.266242 1627 ts_tablet_manager.cc:1918] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:20:24.266389 1629 ts_tablet_manager.cc:1939] T 069f3e6d46554596806ca6198d0087e5 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting consensus metadata
I20260811 22:20:24.266443 1623 raft_consensus.cc:2258] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:24.266609 1627 log.cc:1196] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/069f3e6d46554596806ca6198d0087e5
I20260811 22:20:24.267102 536 catalog_manager.cc:4841] TS 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553): tablet 069f3e6d46554596806ca6198d0087e5 (table test-table [id=f5e77d5cb24945c49c86e9d1c007130f]) successfully deleted
I20260811 22:20:24.267124 1623 ts_tablet_manager.cc:1905] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:24.269825 1623 ts_tablet_manager.cc:1918] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:20:24.269893 1623 log.cc:1196] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/069f3e6d46554596806ca6198d0087e5
I20260811 22:20:24.270107 1623 ts_tablet_manager.cc:1939] T 069f3e6d46554596806ca6198d0087e5 P 607183375abf4bc7a960879b63d0c51d: Deleting consensus metadata
I20260811 22:20:24.270584 1627 ts_tablet_manager.cc:1939] T 069f3e6d46554596806ca6198d0087e5 P b43e7a7b15724a8cada5707a87f3c5de: Deleting consensus metadata
I20260811 22:20:24.270963 539 catalog_manager.cc:4841] TS 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815): tablet 069f3e6d46554596806ca6198d0087e5 (table test-table [id=f5e77d5cb24945c49c86e9d1c007130f]) successfully deleted
I20260811 22:20:24.271343 540 catalog_manager.cc:4841] TS b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67:40613): tablet 069f3e6d46554596806ca6198d0087e5 (table test-table [id=f5e77d5cb24945c49c86e9d1c007130f]) successfully deleted
Aug 11 22:20:24 dist-test-slave-4tdj krb5kdc[479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:20:24 dist-test-slave-4tdj krb5kdc[479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486824, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:24.291716 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:24.282988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39522 (local address 127.29.3.126:38535)
0811 22:20:24.283336 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:24.283343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:24.283359 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:20:24.283434 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:24.283436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:24.283483 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:24.283568 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:24.284368 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:24.284937 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:24.285802 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:24.286048 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:24.286146 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:24.288673 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:24.288677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:24.288679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:24.288726 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:24.290103 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:24.290575 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:24.290578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:24.290579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:24.290662 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:24.290883 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:24.290886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:24.290888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:24.291106 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:24.291221 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:24.291405 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 22:20:24.291533 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
I20260811 22:20:24.294387 562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:24.294837 562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:24.299296 795 tablet_service.cc:1467] Processing CreateTablet for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 (DEFAULT_TABLE table=test-table [id=688f9cd41ec9440e89e073ada306dcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:24.299404 1045 tablet_service.cc:1467] Processing CreateTablet for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 (DEFAULT_TABLE table=test-table [id=688f9cd41ec9440e89e073ada306dcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:24.299525 1306 tablet_service.cc:1467] Processing CreateTablet for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 (DEFAULT_TABLE table=test-table [id=688f9cd41ec9440e89e073ada306dcfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:24.299623 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:24.299690 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:24.299773 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:24.302248 1647 tablet_bootstrap.cc:492] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Bootstrap starting.
I20260811 22:20:24.302316 1648 tablet_bootstrap.cc:492] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Bootstrap starting.
I20260811 22:20:24.303186 1647 tablet_bootstrap.cc:654] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:24.303210 1648 tablet_bootstrap.cc:654] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:24.303741 1649 tablet_bootstrap.cc:492] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Bootstrap starting.
I20260811 22:20:24.304375 1648 tablet_bootstrap.cc:492] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: No bootstrap required, opened a new log
I20260811 22:20:24.304466 1648 ts_tablet_manager.cc:1397] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:24.304564 1647 tablet_bootstrap.cc:492] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: No bootstrap required, opened a new log
I20260811 22:20:24.304694 1647 ts_tablet_manager.cc:1397] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 22:20:24.304749 1649 tablet_bootstrap.cc:654] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:24.304895 1648 raft_consensus.cc:348] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.305040 1648 raft_consensus.cc:374] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:24.305109 1648 raft_consensus.cc:729] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Initialized, Role: FOLLOWER
I20260811 22:20:24.305294 1648 consensus_queue.cc:260] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.305370 1647 raft_consensus.cc:348] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.305578 1647 raft_consensus.cc:374] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:24.305631 1647 raft_consensus.cc:729] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cecbe73fc5b459786221ec4ff0b6bd2, State: Initialized, Role: FOLLOWER
I20260811 22:20:24.305752 1647 consensus_queue.cc:260] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.306088 1648 ts_tablet_manager.cc:1428] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:24.306190 1647 ts_tablet_manager.cc:1428] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:24.306545 1649 tablet_bootstrap.cc:492] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: No bootstrap required, opened a new log
I20260811 22:20:24.306630 1649 ts_tablet_manager.cc:1397] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:24.307114 1649 raft_consensus.cc:348] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.307235 1649 raft_consensus.cc:374] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:24.307276 1649 raft_consensus.cc:729] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607183375abf4bc7a960879b63d0c51d, State: Initialized, Role: FOLLOWER
I20260811 22:20:24.307392 1649 consensus_queue.cc:260] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.307716 1649 ts_tablet_manager.cc:1428] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:24.326758 1614 raft_consensus.cc:482] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:24.326906 1614 raft_consensus.cc:504] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.327335 1614 leader_election.cc:288] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815), 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:24.327704 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673e6d1c4b748f3bfcdfa4dd7ddd409" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d" is_pre_election: true
I20260811 22:20:24.327716 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673e6d1c4b748f3bfcdfa4dd7ddd409" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" is_pre_election: true
I20260811 22:20:24.327839 828 raft_consensus.cc:2454] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:24.327857 1078 raft_consensus.cc:2454] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:24.328141 1210 leader_election.cc:302] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cecbe73fc5b459786221ec4ff0b6bd2, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:24.328310 1614 raft_consensus.cc:2781] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:24.328372 1614 raft_consensus.cc:482] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:24.328406 1614 raft_consensus.cc:3037] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:24.329387 1614 raft_consensus.cc:504] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.329679 1614 leader_election.cc:288] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Requested vote from peers 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815), 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:24.330004 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673e6d1c4b748f3bfcdfa4dd7ddd409" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
I20260811 22:20:24.330020 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f673e6d1c4b748f3bfcdfa4dd7ddd409" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d"
I20260811 22:20:24.330122 1078 raft_consensus.cc:3037] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:24.330147 828 raft_consensus.cc:3037] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:24.331106 1078 raft_consensus.cc:2454] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:24.331251 828 raft_consensus.cc:2454] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:24.331389 1210 leader_election.cc:302] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cecbe73fc5b459786221ec4ff0b6bd2, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:24.331573 1614 raft_consensus.cc:2781] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:24.331681 1614 raft_consensus.cc:686] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Becoming Leader. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Running, Role: LEADER
I20260811 22:20:24.331835 1614 consensus_queue.cc:237] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.333441 563 catalog_manager.cc:5495] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de reported cstate change: term changed from 0 to 1, leader changed from <none> to b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67). New cstate: current_term: 1 leader_uuid: "b43e7a7b15724a8cada5707a87f3c5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } health_report { overall_health: HEALTHY } } }
I20260811 22:20:24.353820 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:24.350750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.29.3.67:40613)
0811 22:20:24.351026 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:24.351033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:24.351044 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:20:24.351082 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:24.351084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:24.351125 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:24.351211 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:24.351472 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:24.351987 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:24.352820 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:24.353043 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:24.353129 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:24.353383 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:24.353456 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:24.353512 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:24.353616 (+ 104us) server_negotiation.cc:299] Negotiation successful
0811 22:20:24.353670 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260811 22:20:24.360046 1078 raft_consensus.cc:1261] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Refusing update from remote peer b43e7a7b15724a8cada5707a87f3c5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:24.360075 828 raft_consensus.cc:1261] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Refusing update from remote peer b43e7a7b15724a8cada5707a87f3c5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:24.360433 1614 consensus_queue.cc:1035] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:24.360594 1582 consensus_queue.cc:1035] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:24.390347 562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:24.390533 562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:24.392516 562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Sending DeleteTablet for 3 replicas of tablet f673e6d1c4b748f3bfcdfa4dd7ddd409
I20260811 22:20:24.393013 795 tablet_service.cc:1514] Processing DeleteTablet for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:24 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260811 22:20:24.393129 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:24 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39132
I20260811 22:20:24.393208 1623 tablet_replica.cc:330] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: stopping tablet replica
I20260811 22:20:24.393180 1306 tablet_service.cc:1514] Processing DeleteTablet for tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:24 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:53652
I20260811 22:20:24.393311 1629 tablet_replica.cc:330] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: stopping tablet replica
I20260811 22:20:24.393323 1623 raft_consensus.cc:2229] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:24.393350 1627 tablet_replica.cc:330] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: stopping tablet replica
I20260811 22:20:24.393401 1629 raft_consensus.cc:2229] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:24.393438 1623 raft_consensus.cc:2258] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:24.393469 1627 raft_consensus.cc:2229] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:24.393497 1629 raft_consensus.cc:2258] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:24.393672 1627 raft_consensus.cc:2258] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:24.393772 562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:24.394170 562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:24.394235 1629 ts_tablet_manager.cc:1905] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:24.394241 1623 ts_tablet_manager.cc:1905] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:24.394526 1627 ts_tablet_manager.cc:1905] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:24.396752 1629 ts_tablet_manager.cc:1918] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:24.396752 1623 ts_tablet_manager.cc:1918] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:24.396822 1623 log.cc:1196] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/f673e6d1c4b748f3bfcdfa4dd7ddd409
I20260811 22:20:24.396821 1629 log.cc:1196] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/f673e6d1c4b748f3bfcdfa4dd7ddd409
I20260811 22:20:24.397013 1627 ts_tablet_manager.cc:1918] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:24.397065 1629 ts_tablet_manager.cc:1939] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting consensus metadata
I20260811 22:20:24.397065 1623 ts_tablet_manager.cc:1939] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P 607183375abf4bc7a960879b63d0c51d: Deleting consensus metadata
I20260811 22:20:24.397084 1627 log.cc:1196] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/f673e6d1c4b748f3bfcdfa4dd7ddd409
I20260811 22:20:24.397337 1627 ts_tablet_manager.cc:1939] T f673e6d1c4b748f3bfcdfa4dd7ddd409 P b43e7a7b15724a8cada5707a87f3c5de: Deleting consensus metadata
I20260811 22:20:24.398075 540 catalog_manager.cc:4841] TS b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67:40613): tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 (table test-table [id=688f9cd41ec9440e89e073ada306dcfd]) successfully deleted
I20260811 22:20:24.398800 536 catalog_manager.cc:4841] TS 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553): tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 (table test-table [id=688f9cd41ec9440e89e073ada306dcfd]) successfully deleted
I20260811 22:20:24.399034 539 catalog_manager.cc:4841] TS 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815): tablet f673e6d1c4b748f3bfcdfa4dd7ddd409 (table test-table [id=688f9cd41ec9440e89e073ada306dcfd]) successfully deleted
I20260811 22:20:24.399152 1045 tablet_service.cc:1467] Processing CreateTablet for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01 (DEFAULT_TABLE table=test-table [id=e365a8fabd204b51babb73cf7fde8fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:24.399227 795 tablet_service.cc:1467] Processing CreateTablet for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01 (DEFAULT_TABLE table=test-table [id=e365a8fabd204b51babb73cf7fde8fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:24.399431 1045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:24.399382 1306 tablet_service.cc:1467] Processing CreateTablet for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01 (DEFAULT_TABLE table=test-table [id=e365a8fabd204b51babb73cf7fde8fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:24.399449 795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:24.399614 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:24.401280 1648 tablet_bootstrap.cc:492] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Bootstrap starting.
I20260811 22:20:24.401669 1647 tablet_bootstrap.cc:492] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: Bootstrap starting.
I20260811 22:20:24.401679 1649 tablet_bootstrap.cc:492] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d: Bootstrap starting.
I20260811 22:20:24.402082 1648 tablet_bootstrap.cc:654] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:24.402464 1649 tablet_bootstrap.cc:654] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:24.402479 1647 tablet_bootstrap.cc:654] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:24.403209 1648 tablet_bootstrap.cc:492] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: No bootstrap required, opened a new log
I20260811 22:20:24.403306 1648 ts_tablet_manager.cc:1397] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:24.403661 1647 tablet_bootstrap.cc:492] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: No bootstrap required, opened a new log
I20260811 22:20:24.403745 1647 ts_tablet_manager.cc:1397] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:24.403685 1648 raft_consensus.cc:348] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.403795 1648 raft_consensus.cc:374] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:24.403836 1648 raft_consensus.cc:729] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Initialized, Role: FOLLOWER
I20260811 22:20:24.403970 1648 consensus_queue.cc:260] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.404237 1648 ts_tablet_manager.cc:1428] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:24.404188 1647 raft_consensus.cc:348] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.404304 1647 raft_consensus.cc:374] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:24.404346 1647 raft_consensus.cc:729] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cecbe73fc5b459786221ec4ff0b6bd2, State: Initialized, Role: FOLLOWER
I20260811 22:20:24.404461 1647 consensus_queue.cc:260] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.404706 1647 ts_tablet_manager.cc:1428] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:24.405189 1649 tablet_bootstrap.cc:492] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d: No bootstrap required, opened a new log
I20260811 22:20:24.405285 1649 ts_tablet_manager.cc:1397] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260811 22:20:24.405663 1649 raft_consensus.cc:348] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.405777 1649 raft_consensus.cc:374] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:24.405813 1649 raft_consensus.cc:729] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607183375abf4bc7a960879b63d0c51d, State: Initialized, Role: FOLLOWER
I20260811 22:20:24.405943 1649 consensus_queue.cc:260] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.406203 1649 ts_tablet_manager.cc:1428] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:24.439467 1604 raft_consensus.cc:482] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:24.439602 1604 raft_consensus.cc:504] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.440001 1604 leader_election.cc:288] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815), 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:24.440404 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f6a622d5dc4fd9ad7e77223e9d2e01" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" is_pre_election: true
I20260811 22:20:24.440407 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f6a622d5dc4fd9ad7e77223e9d2e01" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d" is_pre_election: true
I20260811 22:20:24.440551 828 raft_consensus.cc:2454] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:24.440552 1078 raft_consensus.cc:2454] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 0.
I20260811 22:20:24.440871 1213 leader_election.cc:302] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607183375abf4bc7a960879b63d0c51d, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:24.441051 1604 raft_consensus.cc:2781] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:24.441112 1604 raft_consensus.cc:482] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:24.441167 1604 raft_consensus.cc:3037] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:24.442200 1604 raft_consensus.cc:504] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.442550 1604 leader_election.cc:288] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Requested vote from peers 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815), 7cecbe73fc5b459786221ec4ff0b6bd2 (127.29.3.66:43553)
I20260811 22:20:24.442878 1078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f6a622d5dc4fd9ad7e77223e9d2e01" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2"
I20260811 22:20:24.442865 828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f6a622d5dc4fd9ad7e77223e9d2e01" candidate_uuid: "b43e7a7b15724a8cada5707a87f3c5de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607183375abf4bc7a960879b63d0c51d"
I20260811 22:20:24.442986 1078 raft_consensus.cc:3037] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:24.442992 828 raft_consensus.cc:3037] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:24.443868 828 raft_consensus.cc:2454] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:24.443871 1078 raft_consensus.cc:2454] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b43e7a7b15724a8cada5707a87f3c5de in term 1.
I20260811 22:20:24.444149 1210 leader_election.cc:302] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cecbe73fc5b459786221ec4ff0b6bd2, b43e7a7b15724a8cada5707a87f3c5de; no voters:
I20260811 22:20:24.444319 1604 raft_consensus.cc:2781] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:24.444434 1604 raft_consensus.cc:686] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Becoming Leader. State: Replica: b43e7a7b15724a8cada5707a87f3c5de, State: Running, Role: LEADER
I20260811 22:20:24.444574 1604 consensus_queue.cc:237] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } }
I20260811 22:20:24.446343 562 catalog_manager.cc:5495] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de reported cstate change: term changed from 0 to 1, leader changed from <none> to b43e7a7b15724a8cada5707a87f3c5de (127.29.3.67). New cstate: current_term: 1 leader_uuid: "b43e7a7b15724a8cada5707a87f3c5de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b43e7a7b15724a8cada5707a87f3c5de" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40613 } health_report { overall_health: HEALTHY } } }
I20260811 22:20:24.469620 828 raft_consensus.cc:1261] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 607183375abf4bc7a960879b63d0c51d [term 1 FOLLOWER]: Refusing update from remote peer b43e7a7b15724a8cada5707a87f3c5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:24.469620 1078 raft_consensus.cc:1261] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Refusing update from remote peer b43e7a7b15724a8cada5707a87f3c5de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:24.470041 1582 consensus_queue.cc:1035] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "607183375abf4bc7a960879b63d0c51d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:24.470232 1604 consensus_queue.cc:1035] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cecbe73fc5b459786221ec4ff0b6bd2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 43553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:20:24.483027 1588 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 22:20:26.036639 562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:26.036823 562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:26.039533 562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 510781c0c3cf4275aec8eba21a2b3b92: Sending DeleteTablet for 3 replicas of tablet 12f6a622d5dc4fd9ad7e77223e9d2e01
I20260811 22:20:26.040297 795 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:26 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44890
I20260811 22:20:26.040342 1045 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:26 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39132
I20260811 22:20:26.040380 1306 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:26 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:53652
I20260811 22:20:26.040812 1725 tablet_replica.cc:330] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: stopping tablet replica
I20260811 22:20:26.040947 1725 raft_consensus.cc:2229] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:26.041080 1725 raft_consensus.cc:2258] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:26.041134 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 688
I20260811 22:20:26.051918 1725 ts_tablet_manager.cc:1905] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P 7cecbe73fc5b459786221ec4ff0b6bd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:26.054580 1727 tablet_replica.cc:330] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: stopping tablet replica
I20260811 22:20:26.054730 1727 raft_consensus.cc:2229] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:26.054962 1727 raft_consensus.cc:2258] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:26.055636 1727 ts_tablet_manager.cc:1905] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 22:20:26.065243 539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:32815 (error 108)
I20260811 22:20:26.067857 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 923
I20260811 22:20:26.078284 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:26.067687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:32815 (local address 127.0.0.1:44916)
0811 22:20:26.077980 (+ 10293us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:26.078141 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
W20260811 22:20:26.078493 539 catalog_manager.cc:4584] TS 607183375abf4bc7a960879b63d0c51d (127.29.3.65:32815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12f6a622d5dc4fd9ad7e77223e9d2e01: Network error: Client connection negotiation failed: client connection to 127.29.3.65:32815: connect: Connection refused (error 111)
I20260811 22:20:26.080067 1727 ts_tablet_manager.cc:1918] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 22:20:26.080150 1727 log.cc:1196] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/12f6a622d5dc4fd9ad7e77223e9d2e01
I20260811 22:20:26.080433 1727 ts_tablet_manager.cc:1939] T 12f6a622d5dc4fd9ad7e77223e9d2e01 P b43e7a7b15724a8cada5707a87f3c5de: Deleting consensus metadata
I20260811 22:20:26.090880 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1178
I20260811 22:20:26.097836 1729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:26.090860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:43553 (local address 127.0.0.1:39146)
0811 22:20:26.097635 (+ 6775us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:26.097724 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.66:43553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 22:20:26.118517 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 509
2026-08-11T22:20:26Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6818 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:26 dist-test-slave-4tdj krb5kdc[1733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:26 dist-test-slave-4tdj krb5kdc[1733](info): set up 2 sockets
Aug 11 22:20:26 dist-test-slave-4tdj krb5kdc[1733](info): commencing operation
krb5kdc: starting...
W20260811 22:20:27.851519 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.693s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:27 dist-test-slave-4tdj krb5kdc[1733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:27.977643 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:34853
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:34853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:28.091699 1752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:28.092008 1752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:28.092048 1752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:28.096077 1752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:28.096134 1752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:28.096153 1752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:28.096167 1752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:28.096181 1752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:28.101475 1752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:34853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:34853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:28.102587 1752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:28.103623 1752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:28.112567 1764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.112615 1768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.112672 1762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:28.112831 1752 server_base.cc:1034] running on GCE node
I20260811 22:20:28.113431 1752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:28.114606 1752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:28.115790 1752 hybrid_clock.cc:648] HybridClock initialized: now 1786486828115775 us; error 25 us; skew 500 ppm
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:28.118510 1752 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:28.119441 1752 webserver.cc:460] Webserver started at http://127.29.3.126:34439/ using document root <none> and password file <none>
I20260811 22:20:28.119989 1752 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:28.120057 1752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:28.120270 1752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:28.122375 1752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "802ef4e9dfa14f59867569761dfda0cc"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "d14edb180a8b7bca26f57fd2e6b521cc"
server_key_iv: "37aab87b83c2b3aef90daecbad3fdffa"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.122933 1752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "802ef4e9dfa14f59867569761dfda0cc"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "d14edb180a8b7bca26f57fd2e6b521cc"
server_key_iv: "37aab87b83c2b3aef90daecbad3fdffa"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.126664 1752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260811 22:20:28.129019 1782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.130219 1752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:28.130378 1752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "802ef4e9dfa14f59867569761dfda0cc"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "d14edb180a8b7bca26f57fd2e6b521cc"
server_key_iv: "37aab87b83c2b3aef90daecbad3fdffa"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.130479 1752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:28.153152 1752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:28.156468 1752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:28.156661 1752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:28.163862 1752 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:34853
I20260811 22:20:28.163879 1870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:34853 every 8 connection(s)
I20260811 22:20:28.164846 1752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:28.167654 1871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:28.173136 1871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc: Bootstrap starting.
I20260811 22:20:28.174829 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1752
I20260811 22:20:28.174944 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:28.175230 29709 external_mini_cluster.cc:1428] Setting key fb64f13220a151e00cdf55f8cc9f0be6
I20260811 22:20:28.175870 1871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:28.176754 1871 log.cc:824] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:28.178870 1871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc: No bootstrap required, opened a new log
I20260811 22:20:28.182943 1871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802ef4e9dfa14f59867569761dfda0cc" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34853 } }
I20260811 22:20:28.183229 1871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:28.183300 1871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 802ef4e9dfa14f59867569761dfda0cc, State: Initialized, Role: FOLLOWER
I20260811 22:20:28.183753 1871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802ef4e9dfa14f59867569761dfda0cc" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34853 } }
I20260811 22:20:28.183880 1871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:28.183940 1871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:28.184044 1871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 0 FOLLOWER]: Advancing to term 1
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:28.185010 1871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802ef4e9dfa14f59867569761dfda0cc" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34853 } }
I20260811 22:20:28.185402 1871 leader_election.cc:302] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 802ef4e9dfa14f59867569761dfda0cc; no voters:
I20260811 22:20:28.185782 1871 leader_election.cc:288] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:28.185930 1879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:28.186112 1879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [term 1 LEADER]: Becoming Leader. State: Replica: 802ef4e9dfa14f59867569761dfda0cc, State: Running, Role: LEADER
I20260811 22:20:28.186460 1879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802ef4e9dfa14f59867569761dfda0cc" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34853 } }
I20260811 22:20:28.187132 1871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:28.187660 1881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 802ef4e9dfa14f59867569761dfda0cc. Latest consensus state: current_term: 1 leader_uuid: "802ef4e9dfa14f59867569761dfda0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802ef4e9dfa14f59867569761dfda0cc" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34853 } } }
I20260811 22:20:28.187781 1881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:28.188424 1882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "802ef4e9dfa14f59867569761dfda0cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "802ef4e9dfa14f59867569761dfda0cc" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34853 } } }
I20260811 22:20:28.188545 1882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:28.189692 1890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:28.190971 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.176359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33082 (local address 127.29.3.126:34853)
0811 22:20:28.176769 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:28.176785 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:28.176824 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:20:28.177591 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:28.177613 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:28.177951 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:28.178232 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:28.179116 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.180081 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.181256 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.181702 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.181823 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.184803 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:28.184810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:28.184827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:28.184861 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:28.186764 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.187312 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.187317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.187323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.187481 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.187926 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.187930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.187932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.188482 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:28.188689 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:28.189615 (+ 926us) server_negotiation.cc:299] Negotiation successful
0811 22:20:28.189935 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260811 22:20:28.192706 1890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:28.198364 1890 catalog_manager.cc:1347] Generated new cluster ID: 2c64b53c89f649c9802f230266d201ee
I20260811 22:20:28.198451 1890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:28.208088 1890 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:28.209321 1890 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:28.216836 1890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 802ef4e9dfa14f59867569761dfda0cc: Generated new TSK 0
I20260811 22:20:28.217643 1890 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:28.247192 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34853
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:28.360210 1919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:28.360512 1919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:28.360572 1919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:28.364759 1919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:28.364833 1919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:28.364908 1919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:28.370316 1919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:28.371440 1919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:28.372509 1919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:28.381978 1929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.382066 1934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.382016 1932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:28.382237 1919 server_base.cc:1034] running on GCE node
I20260811 22:20:28.382645 1919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:28.383275 1919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:28.384428 1919 hybrid_clock.cc:648] HybridClock initialized: now 1786486828384411 us; error 31 us; skew 500 ppm
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:28.387187 1919 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:28.388149 1919 webserver.cc:460] Webserver started at http://127.29.3.65:43613/ using document root <none> and password file <none>
I20260811 22:20:28.388691 1919 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:28.388760 1919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:28.388967 1919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:28.391031 1919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "5cf80d504f544ca58572af9c41a9aee4"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "486b1d26714c6cad5245e525f4902ab6"
server_key_iv: "004646f8aa7f8d0689c6133a70c579a1"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.391605 1919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "5cf80d504f544ca58572af9c41a9aee4"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "486b1d26714c6cad5245e525f4902ab6"
server_key_iv: "004646f8aa7f8d0689c6133a70c579a1"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.395279 1919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:20:28.397594 1957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.398825 1919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:28.398933 1919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "5cf80d504f544ca58572af9c41a9aee4"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "486b1d26714c6cad5245e525f4902ab6"
server_key_iv: "004646f8aa7f8d0689c6133a70c579a1"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.399027 1919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:28.411471 1919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:28.414610 1919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:28.414803 1919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:28.415390 1919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:28.416244 1919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:28.416296 1919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.416348 1919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:28.416378 1919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.425127 1919 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:35637
I20260811 22:20:28.425179 2145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:35637 every 8 connection(s)
I20260811 22:20:28.426082 1919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:28.433904 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1919
I20260811 22:20:28.434016 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:28.434356 29709 external_mini_cluster.cc:1428] Setting key 6241370c5b664687786fcf0fdeba009c
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:28.439803 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.427947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:45123 (local address 127.29.3.126:34853)
0811 22:20:28.428084 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:28.428088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:28.428918 (+ 830us) server_negotiation.cc:407] Connection header received
0811 22:20:28.430320 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:28.430323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:28.430384 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:28.430453 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:28.431637 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.432168 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.433304 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.433530 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.433619 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.436371 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:28.436374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:28.436376 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:28.436399 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:28.437780 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.438271 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.438275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.438277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.438408 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.438707 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.438709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.438719 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.438943 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:28.439046 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:28.439516 (+ 470us) server_negotiation.cc:299] Negotiation successful
0811 22:20:28.439639 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:28.440587 2151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.428325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:34853 (local address 127.29.3.65:45123)
0811 22:20:28.428774 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:28.428819 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:28.430057 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:28.430640 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:28.430647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:28.430996 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:28.431490 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:28.431504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.432292 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.432295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:28.433177 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:28.433188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.433697 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.433702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:28.433858 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.434525 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:28.434545 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:28.436208 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:28.437911 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:28.437916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:28.437929 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:28.438186 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:28.438488 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:28.438490 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:28.438491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:28.438625 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:28.439030 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:28.439035 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:28.439284 (+ 249us) client_negotiation.cc:769] Sending connection context
0811 22:20:28.439438 (+ 154us) client_negotiation.cc:240] Negotiation successful
0811 22:20:28.439696 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260811 22:20:28.441733 2149 heartbeater.cc:347] Connected to a master server at 127.29.3.126:34853
I20260811 22:20:28.441965 2149 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:28.442477 2149 heartbeater.cc:511] Master 127.29.3.126:34853 requested a full tablet report, sending...
I20260811 22:20:28.443961 1807 ts_manager.cc:194] Registered new tserver with Master: 5cf80d504f544ca58572af9c41a9aee4 (127.29.3.65:35637)
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:20:28.445088 1807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:45123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:28.462930 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34853
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:28.578567 2161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:28.578833 2161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:28.578895 2161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:28.583105 2161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:28.583194 2161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:28.583281 2161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:28.588922 2161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:28.590111 2161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:28.591192 2161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:28.600943 2178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.600960 2176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.601074 2180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:28.601212 2161 server_base.cc:1034] running on GCE node
I20260811 22:20:28.601744 2161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:28.602451 2161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:28.603628 2161 hybrid_clock.cc:648] HybridClock initialized: now 1786486828603614 us; error 38 us; skew 500 ppm
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:28.606941 2161 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:28.607978 2161 webserver.cc:460] Webserver started at http://127.29.3.66:34457/ using document root <none> and password file <none>
I20260811 22:20:28.608557 2161 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:28.608619 2161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:28.608822 2161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:28.611085 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "abbc60c445ed4f8886ba09d955cdc110"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "cd386ce0429bf942c7bdeb9cdf3ccb37"
server_key_iv: "97dd1fcfcf28430b365c1d6d8889d595"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.611675 2161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "abbc60c445ed4f8886ba09d955cdc110"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "cd386ce0429bf942c7bdeb9cdf3ccb37"
server_key_iv: "97dd1fcfcf28430b365c1d6d8889d595"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.615605 2161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:20:28.618081 2192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.619408 2161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:28.619516 2161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "abbc60c445ed4f8886ba09d955cdc110"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "cd386ce0429bf942c7bdeb9cdf3ccb37"
server_key_iv: "97dd1fcfcf28430b365c1d6d8889d595"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.619629 2161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:28.667260 2161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:28.670560 2161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:28.670786 2161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:28.671435 2161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:28.672318 2161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:28.672365 2161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.672406 2161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:28.672434 2161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.682473 2161 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:34765
I20260811 22:20:28.682519 2403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:34765 every 8 connection(s)
I20260811 22:20:28.683415 2161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:28.691425 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2161
I20260811 22:20:28.691519 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:28.691783 29709 external_mini_cluster.cc:1428] Setting key e71246ca68b1d368ed97c1b6f516e11d
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:28.700125 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.685271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:51871 (local address 127.29.3.126:34853)
0811 22:20:28.685443 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:28.685448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:28.686424 (+ 976us) server_negotiation.cc:407] Connection header received
0811 22:20:28.688364 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:28.688367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:28.688419 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:28.688496 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:28.690212 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.690791 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.692817 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.693074 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.693156 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.696173 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:28.696176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:28.696178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:28.696204 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:28.697716 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.698392 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.698395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.698397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.698496 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.698930 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.698934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.698936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.699189 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:28.699291 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:28.699835 (+ 544us) server_negotiation.cc:299] Negotiation successful
0811 22:20:28.699961 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:20:28.701061 2411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.685653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:34853 (local address 127.29.3.66:51871)
0811 22:20:28.686216 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:28.686279 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:28.688068 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:28.688720 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:28.688730 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:28.689287 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:28.690046 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:28.690066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.690925 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.690929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:28.692670 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:28.692684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.693233 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.693238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:28.693460 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.694150 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:28.694181 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:28.695999 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:28.697854 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:28.697861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:28.697879 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:28.698277 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:28.698601 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:28.698605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:28.698607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:28.698818 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:28.699292 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:28.699298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:28.699570 (+ 272us) client_negotiation.cc:769] Sending connection context
0811 22:20:28.699753 (+ 183us) client_negotiation.cc:240] Negotiation successful
0811 22:20:28.700101 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":110,"threads_started":1}
I20260811 22:20:28.702437 2405 heartbeater.cc:347] Connected to a master server at 127.29.3.126:34853
I20260811 22:20:28.702741 2405 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:28.703342 2405 heartbeater.cc:511] Master 127.29.3.126:34853 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:20:28.704450 1807 ts_manager.cc:194] Registered new tserver with Master: abbc60c445ed4f8886ba09d955cdc110 (127.29.3.66:34765)
I20260811 22:20:28.705114 1807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:51871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:28.725030 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34853
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:28.843401 2419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:28.843665 2419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:28.843712 2419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:28.847715 2419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:28.847777 2419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:28.847851 2419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:28.853097 2419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:28.854174 2419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:28.855230 2419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:28.864928 2433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:28.865037 2419 server_base.cc:1034] running on GCE node
W20260811 22:20:28.865060 2429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:28.865038 2428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:28.865651 2419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:28.866276 2419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:28.867487 2419 hybrid_clock.cc:648] HybridClock initialized: now 1786486828867456 us; error 40 us; skew 500 ppm
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:28.870529 2419 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:28.871415 2419 webserver.cc:460] Webserver started at http://127.29.3.67:37513/ using document root <none> and password file <none>
I20260811 22:20:28.872037 2419 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:28.872107 2419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:28.872316 2419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:28.874425 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "61f7784bd30347e7803049f2f61a291c"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "45b8cc10be4bc27f4f09509ccdaa53fc"
server_key_iv: "dff5d106497af10ad1d3fe24c60b6e11"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.874974 2419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "61f7784bd30347e7803049f2f61a291c"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "45b8cc10be4bc27f4f09509ccdaa53fc"
server_key_iv: "dff5d106497af10ad1d3fe24c60b6e11"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.878865 2419 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:20:28.881424 2443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.882768 2419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:28.882884 2419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "61f7784bd30347e7803049f2f61a291c"
format_stamp: "Formatted at 2026-08-11 22:20:28 on dist-test-slave-4tdj"
server_key: "45b8cc10be4bc27f4f09509ccdaa53fc"
server_key_iv: "dff5d106497af10ad1d3fe24c60b6e11"
server_key_version: "encryptionkey@0"
I20260811 22:20:28.882987 2419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:28.908056 2419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:28.911222 2419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:28.911406 2419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:28.912024 2419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:28.912900 2419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:28.912950 2419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.913005 2419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:28.913035 2419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:28.921424 2419 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:36725
I20260811 22:20:28.921450 2678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:36725 every 8 connection(s)
I20260811 22:20:28.922329 2419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:28.922670 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2419
I20260811 22:20:28.922787 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:28.923110 29709 external_mini_cluster.cc:1428] Setting key 6f92e63a9461e85565237ab6e78079d6
Aug 11 22:20:28 dist-test-slave-4tdj krb5kdc[1733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486828, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:28.935702 1875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.924334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:49095 (local address 127.29.3.126:34853)
0811 22:20:28.924467 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:28.924471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:28.925478 (+ 1007us) server_negotiation.cc:407] Connection header received
0811 22:20:28.926833 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:28.926836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:28.926889 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:28.926960 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:28.928112 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.928654 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.929758 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.929969 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.930056 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.932370 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:28.932373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:28.932375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:28.932400 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:28.933739 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.934212 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.934215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.934216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.934312 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:28.934650 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:28.934653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:28.934655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:28.934829 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:28.934913 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:28.935409 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 22:20:28.935527 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:28.936643 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:28.924791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:34853 (local address 127.29.3.67:49095)
0811 22:20:28.925328 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:28.925383 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:28.926642 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:28.927164 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:28.927172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:28.927510 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:28.927974 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:28.927987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.928770 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.928772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:28.929643 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:28.929652 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:28.930111 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:28.930115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:28.930271 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:28.930807 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:28.930823 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:28.932215 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:28.933845 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:28.933851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:28.933867 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:28.934118 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:28.934417 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:28.934420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:28.934422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:28.934561 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:28.934926 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:28.934933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:28.935184 (+ 251us) client_negotiation.cc:769] Sending connection context
0811 22:20:28.935350 (+ 166us) client_negotiation.cc:240] Negotiation successful
0811 22:20:28.935674 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":125,"threads_started":1}
I20260811 22:20:28.937714 2681 heartbeater.cc:347] Connected to a master server at 127.29.3.126:34853
I20260811 22:20:28.937937 2681 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:28.938453 2681 heartbeater.cc:511] Master 127.29.3.126:34853 requested a full tablet report, sending...
I20260811 22:20:28.939435 1807 ts_manager.cc:194] Registered new tserver with Master: 61f7784bd30347e7803049f2f61a291c (127.29.3.67:36725)
I20260811 22:20:28.939996 1807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:49095
I20260811 22:20:28.949055 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 22:20:28.963006 1875 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:28.961453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.29.3.126:34853)
0811 22:20:28.961642 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:28.961647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:28.961748 (+ 101us) server_negotiation.cc:407] Connection header received
0811 22:20:28.961898 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:28.961900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:28.961944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:28.962015 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:28.962856 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33086: BlockingRecv error: recv got EOF from 127.0.0.1:33086 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 22:20:28.963886 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1919
I20260811 22:20:28.991302 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2161
I20260811 22:20:29.017771 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2419
I20260811 22:20:29.031806 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1752
2026-08-11T22:20:29Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2903 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:29 dist-test-slave-4tdj krb5kdc[2711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:29 dist-test-slave-4tdj krb5kdc[2711](info): set up 2 sockets
Aug 11 22:20:29 dist-test-slave-4tdj krb5kdc[2711](info): commencing operation
krb5kdc: starting...
W20260811 22:20:30.257503 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:30 dist-test-slave-4tdj krb5kdc[2711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:30.383643 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:33449
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:33449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:30.499141 2735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:30.499423 2735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:30.499480 2735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:30.503585 2735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:30.503659 2735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:30.503686 2735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:30.503707 2735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:30.503746 2735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:30.509061 2735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:33449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:33449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:30.510149 2735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:30.511252 2735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:30.519656 2747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:30.519656 2742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:30.519685 2743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:30.519829 2735 server_base.cc:1034] running on GCE node
I20260811 22:20:30.520533 2735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:30.521620 2735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:30.522787 2735 hybrid_clock.cc:648] HybridClock initialized: now 1786486830522773 us; error 32 us; skew 500 ppm
Aug 11 22:20:30 dist-test-slave-4tdj krb5kdc[2711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:30.525496 2735 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:30.526451 2735 webserver.cc:460] Webserver started at http://127.29.3.126:37681/ using document root <none> and password file <none>
I20260811 22:20:30.526979 2735 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:30.527035 2735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:30.527213 2735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:30.529211 2735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "31b8ce087bdc4b539eabfefb744b5b8f"
format_stamp: "Formatted at 2026-08-11 22:20:30 on dist-test-slave-4tdj"
server_key: "5aab850a664a7cb6a1eac9571a14a1b5"
server_key_iv: "02e578ff81fefa422c68abd44a73a42b"
server_key_version: "encryptionkey@0"
I20260811 22:20:30.529825 2735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "31b8ce087bdc4b539eabfefb744b5b8f"
format_stamp: "Formatted at 2026-08-11 22:20:30 on dist-test-slave-4tdj"
server_key: "5aab850a664a7cb6a1eac9571a14a1b5"
server_key_iv: "02e578ff81fefa422c68abd44a73a42b"
server_key_version: "encryptionkey@0"
I20260811 22:20:30.533614 2735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:20:30.536145 2759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:30.537340 2735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:30.537446 2735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "31b8ce087bdc4b539eabfefb744b5b8f"
format_stamp: "Formatted at 2026-08-11 22:20:30 on dist-test-slave-4tdj"
server_key: "5aab850a664a7cb6a1eac9571a14a1b5"
server_key_iv: "02e578ff81fefa422c68abd44a73a42b"
server_key_version: "encryptionkey@0"
I20260811 22:20:30.537532 2735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:30.556974 2735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:30.560227 2735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:30.560400 2735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:30.567625 2735 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:33449
I20260811 22:20:30.567667 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:33449 every 8 connection(s)
I20260811 22:20:30.568608 2735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:30.570822 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2735
I20260811 22:20:30.570935 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:30.571230 29709 external_mini_cluster.cc:1428] Setting key 7081af204c60569c8bc0e37d303e8b9f
I20260811 22:20:30.571833 2866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:30.578444 2866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f: Bootstrap starting.
Aug 11 22:20:30 dist-test-slave-4tdj krb5kdc[2711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:30.580740 2866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:30.581597 2866 log.cc:824] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:30.583647 2866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f: No bootstrap required, opened a new log
I20260811 22:20:30.585323 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:30.572553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.29.3.126:33449)
0811 22:20:30.572976 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:30.572992 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:30.573028 (+ 36us) server_negotiation.cc:407] Connection header received
0811 22:20:30.573675 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:30.573693 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:30.574014 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:30.574325 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:30.575340 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:30.576347 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:30.577501 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:30.577967 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:30.578079 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:30.580621 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:30.580626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:30.580640 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:30.580678 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:30.582557 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:30.582977 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:30.582983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:30.582989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:30.583102 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:30.583365 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:30.583368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:30.583369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:30.583832 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:30.584008 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:30.584305 (+ 297us) server_negotiation.cc:299] Negotiation successful
0811 22:20:30.584601 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":129,"threads_started":1}
I20260811 22:20:30.587471 2866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33449 } }
I20260811 22:20:30.587738 2866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:30.587802 2866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b8ce087bdc4b539eabfefb744b5b8f, State: Initialized, Role: FOLLOWER
I20260811 22:20:30.588209 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:20:30.588249 2866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33449 } }
I20260811 22:20:30.588402 2866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:30.588465 2866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:30.588559 2866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:30.589444 2866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33449 } }
I20260811 22:20:30.589769 2866 leader_election.cc:302] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b8ce087bdc4b539eabfefb744b5b8f; no voters:
I20260811 22:20:30.590059 2866 leader_election.cc:288] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:30.590157 2875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:30.590446 2875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [term 1 LEADER]: Becoming Leader. State: Replica: 31b8ce087bdc4b539eabfefb744b5b8f, State: Running, Role: LEADER
I20260811 22:20:30.590792 2875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33449 } }
I20260811 22:20:30.591383 2866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:30.592386 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33449 } } }
I20260811 22:20:30.592597 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:30.592859 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b8ce087bdc4b539eabfefb744b5b8f. Latest consensus state: current_term: 1 leader_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b8ce087bdc4b539eabfefb744b5b8f" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33449 } } }
I20260811 22:20:30.593011 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:30.593724 2893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:30.596427 2893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:30.601768 2893 catalog_manager.cc:1347] Generated new cluster ID: 32d4d45fce2e428d97abc68d04e52200
I20260811 22:20:30.601832 2893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:30.612205 2873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:30.606654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.29.3.126:33449)
0811 22:20:30.606790 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:30.606795 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:30.606968 (+ 173us) server_negotiation.cc:407] Connection header received
0811 22:20:30.607090 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:30.607093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:30.607146 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:30.607231 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:30.607778 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:30.608597 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:30.609689 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:30.610069 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:30.610167 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:30.611055 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:30.611059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:30.611062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:30.611633 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:30.611670 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:30.611690 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:30.611774 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:30.611925 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 22:20:30.612054 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260811 22:20:30.616557 2893 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:30.617762 2893 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:30.625608 2893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31b8ce087bdc4b539eabfefb744b5b8f: Generated new TSK 0
I20260811 22:20:30.626505 2893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:20:30.632853 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2735
2026-08-11T22:20:30Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1600 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:30 dist-test-slave-4tdj krb5kdc[2920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:30 dist-test-slave-4tdj krb5kdc[2920](info): set up 2 sockets
Aug 11 22:20:30 dist-test-slave-4tdj krb5kdc[2920](info): commencing operation
krb5kdc: starting...
W20260811 22:20:31.825699 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:31 dist-test-slave-4tdj krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:31.953145 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36265
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:36265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:32.065961 2943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:32.066278 2943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:32.066358 2943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:32.070469 2943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:32.070535 2943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:32.070561 2943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:32.070585 2943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:32.070613 2943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:32.076036 2943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:36265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:32.077098 2943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:32.078116 2943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:32.086632 2958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:32.086665 2943 server_base.cc:1034] running on GCE node
W20260811 22:20:32.086652 2954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.086702 2953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:32.087599 2943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:32.088894 2943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:32.090081 2943 hybrid_clock.cc:648] HybridClock initialized: now 1786486832090063 us; error 35 us; skew 500 ppm
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:32.096331 2943 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:32.097293 2943 webserver.cc:460] Webserver started at http://127.29.3.126:38335/ using document root <none> and password file <none>
I20260811 22:20:32.097860 2943 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:32.097926 2943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:32.098140 2943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:32.100445 2943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "41a15931406344b2ae43cb096a83a613"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "f1876ebc5e076944e91b1b029622d854"
server_key_iv: "af12ce6806a4e919b198c33ab6116e06"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.101016 2943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "41a15931406344b2ae43cb096a83a613"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "f1876ebc5e076944e91b1b029622d854"
server_key_iv: "af12ce6806a4e919b198c33ab6116e06"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.104964 2943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:20:32.107616 2969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.108901 2943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:32.109033 2943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "41a15931406344b2ae43cb096a83a613"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "f1876ebc5e076944e91b1b029622d854"
server_key_iv: "af12ce6806a4e919b198c33ab6116e06"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.109140 2943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:32.120550 2943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:32.123798 2943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:32.123965 2943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:32.131057 2943 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:36265
I20260811 22:20:32.131316 3053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:36265 every 8 connection(s)
I20260811 22:20:32.132063 2943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:32.134956 3054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:32.139947 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2943
I20260811 22:20:32.140057 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:32.140308 29709 external_mini_cluster.cc:1428] Setting key dbad4496742d436ec3313128bc08f27e
I20260811 22:20:32.140558 3054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613: Bootstrap starting.
I20260811 22:20:32.143056 3054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:32.143957 3054 log.cc:824] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:32.146085 3054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613: No bootstrap required, opened a new log
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:32.150264 3054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a15931406344b2ae43cb096a83a613" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36265 } }
I20260811 22:20:32.150539 3054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:32.150600 3054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a15931406344b2ae43cb096a83a613, State: Initialized, Role: FOLLOWER
I20260811 22:20:32.151139 3054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a15931406344b2ae43cb096a83a613" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36265 } }
I20260811 22:20:32.151316 3054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:32.151386 3054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:32.151489 3054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:32.152596 3054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a15931406344b2ae43cb096a83a613" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36265 } }
I20260811 22:20:32.152973 3054 leader_election.cc:302] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a15931406344b2ae43cb096a83a613; no voters:
I20260811 22:20:32.153393 3054 leader_election.cc:288] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:32.153471 3070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:32.153898 3070 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [term 1 LEADER]: Becoming Leader. State: Replica: 41a15931406344b2ae43cb096a83a613, State: Running, Role: LEADER
I20260811 22:20:32.154264 3070 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a15931406344b2ae43cb096a83a613" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36265 } }
I20260811 22:20:32.154770 3054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:32.155001 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.141574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39658 (local address 127.29.3.126:36265)
0811 22:20:32.142008 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:32.142024 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:32.142060 (+ 36us) server_negotiation.cc:407] Connection header received
0811 22:20:32.142747 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:32.142767 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:32.143102 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:32.143420 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:32.144307 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.145308 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.146451 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.146923 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.147041 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.149710 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:32.149716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:32.149730 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:32.149759 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:32.151706 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.152120 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.152126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.152132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.152292 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.152530 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.152534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.152536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.153104 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:32.153311 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:32.153573 (+ 262us) server_negotiation.cc:299] Negotiation successful
0811 22:20:32.153902 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":141,"threads_started":1}
I20260811 22:20:32.156191 3071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a15931406344b2ae43cb096a83a613. Latest consensus state: current_term: 1 leader_uuid: "41a15931406344b2ae43cb096a83a613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a15931406344b2ae43cb096a83a613" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36265 } } }
I20260811 22:20:32.156232 3072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a15931406344b2ae43cb096a83a613" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a15931406344b2ae43cb096a83a613" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36265 } } }
I20260811 22:20:32.156322 3071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:32.156334 3072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:32.156792 3089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:32.160123 3089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:32.165777 3089 catalog_manager.cc:1347] Generated new cluster ID: a5a51050dedb44019bfdedb1e0311bd1
I20260811 22:20:32.165856 3089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:32.176026 3089 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:32.177222 3089 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:32.193468 3089 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a15931406344b2ae43cb096a83a613: Generated new TSK 0
I20260811 22:20:32.194206 3089 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:32.231403 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36265
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:32.365692 3108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:32.365947 3108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:32.365991 3108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:32.370002 3108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:32.370056 3108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:32.370126 3108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:32.375474 3108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:32.376502 3108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:32.377496 3108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:32.386667 3125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.386662 3128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.386662 3132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:32.386906 3108 server_base.cc:1034] running on GCE node
I20260811 22:20:32.387358 3108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:32.387948 3108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:32.389117 3108 hybrid_clock.cc:648] HybridClock initialized: now 1786486832389096 us; error 32 us; skew 500 ppm
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:32.391698 3108 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:32.392509 3108 webserver.cc:460] Webserver started at http://127.29.3.65:35613/ using document root <none> and password file <none>
I20260811 22:20:32.393023 3108 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:32.393082 3108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:32.393296 3108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:32.395324 3108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "63072ba56ee94e6ba7811723db8843b5"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "d210bbf760be8753c94fa3fafa803466"
server_key_iv: "72e0c868de481f9db251c3585fd7ba9a"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.395859 3108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "63072ba56ee94e6ba7811723db8843b5"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "d210bbf760be8753c94fa3fafa803466"
server_key_iv: "72e0c868de481f9db251c3585fd7ba9a"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.399518 3108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:32.401863 3148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.403065 3108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:32.403190 3108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "63072ba56ee94e6ba7811723db8843b5"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "d210bbf760be8753c94fa3fafa803466"
server_key_iv: "72e0c868de481f9db251c3585fd7ba9a"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.403290 3108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:32.430173 3108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:32.433035 3108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:32.433233 3108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:32.433861 3108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:32.434780 3108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:32.434831 3108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.434885 3108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:32.434916 3108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.443476 3108 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:33983
I20260811 22:20:32.443502 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:33983 every 8 connection(s)
I20260811 22:20:32.444351 3108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:32.449096 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3108
I20260811 22:20:32.449200 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:32.449460 29709 external_mini_cluster.cc:1428] Setting key f83a91dd4a94ad79e36589d0d0aa1e4c
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:32.458323 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.446170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:51643 (local address 127.29.3.126:36265)
0811 22:20:32.446332 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:32.446336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:32.447156 (+ 820us) server_negotiation.cc:407] Connection header received
0811 22:20:32.448546 (+ 1390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:32.448550 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:32.448616 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:32.448689 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:32.449916 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.450567 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.451783 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.452089 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.452196 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.454805 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:32.454807 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:32.454809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:32.454835 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:32.456311 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.456814 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.456817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.456819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.456902 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.457221 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.457224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.457226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.457412 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:32.457498 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:32.457993 (+ 495us) server_negotiation.cc:299] Negotiation successful
0811 22:20:32.458122 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:20:32.459148 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.446556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36265 (local address 127.29.3.65:51643)
0811 22:20:32.447011 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:32.447054 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:32.448334 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:32.448876 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:32.448883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:32.449256 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:32.449774 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:32.449789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.450707 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.450711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:32.451639 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:32.451650 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.452223 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.452225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:32.452364 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.452977 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:32.452998 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:32.454630 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:32.456415 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:32.456421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:32.456438 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:32.456721 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:32.456983 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:32.456986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:32.456988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:32.457125 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:32.457511 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:32.457518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:32.457765 (+ 247us) client_negotiation.cc:769] Sending connection context
0811 22:20:32.457935 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 22:20:32.458281 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:20:32.460415 3366 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36265
I20260811 22:20:32.460659 3366 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:32.461158 3366 heartbeater.cc:511] Master 127.29.3.126:36265 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:20:32.462729 2997 ts_manager.cc:194] Registered new tserver with Master: 63072ba56ee94e6ba7811723db8843b5 (127.29.3.65:33983)
I20260811 22:20:32.464175 2997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:51643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:32.480651 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36265
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:32.599511 3374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:32.599779 3374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:32.599843 3374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:32.603875 3374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:32.603943 3374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:32.604025 3374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:32.609323 3374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:32.610431 3374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:32.611454 3374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:32.621969 3383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.622016 3382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.621976 3385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:32.622364 3374 server_base.cc:1034] running on GCE node
I20260811 22:20:32.622753 3374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:32.623392 3374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:32.624578 3374 hybrid_clock.cc:648] HybridClock initialized: now 1786486832624559 us; error 30 us; skew 500 ppm
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:32.627362 3374 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:32.628270 3374 webserver.cc:460] Webserver started at http://127.29.3.66:45269/ using document root <none> and password file <none>
I20260811 22:20:32.628820 3374 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:32.628881 3374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:32.629058 3374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:32.631124 3374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "d9b5faeca9c84d8d8b1ad8d995731628"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "e2d2efe5d49551f8dab29f4237b29585"
server_key_iv: "b6c0d820e07a06cc154f46dadeccfe51"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.631634 3374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "d9b5faeca9c84d8d8b1ad8d995731628"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "e2d2efe5d49551f8dab29f4237b29585"
server_key_iv: "b6c0d820e07a06cc154f46dadeccfe51"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.635275 3374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:32.637701 3405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.639036 3374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:32.639150 3374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "d9b5faeca9c84d8d8b1ad8d995731628"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "e2d2efe5d49551f8dab29f4237b29585"
server_key_iv: "b6c0d820e07a06cc154f46dadeccfe51"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.639238 3374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:32.656447 3374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:32.659407 3374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:32.659590 3374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:32.660193 3374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:32.661051 3374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:32.661100 3374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.661155 3374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:32.661185 3374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.669780 3374 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:44275
I20260811 22:20:32.669811 3602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:44275 every 8 connection(s)
I20260811 22:20:32.670730 3374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:32.677902 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3374
I20260811 22:20:32.677995 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:32.678268 29709 external_mini_cluster.cc:1428] Setting key c8f8c5cffebf7bd2f098b5681d98bfaf
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:32.684444 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.672538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:57847 (local address 127.29.3.126:36265)
0811 22:20:32.672669 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:32.672673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:32.673494 (+ 821us) server_negotiation.cc:407] Connection header received
0811 22:20:32.674916 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:32.674919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:32.674977 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:32.675052 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:32.676235 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.676759 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.677882 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.678142 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.678248 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.680705 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:32.680709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:32.680712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:32.680738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:32.682255 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.682758 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.682762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.682764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.682858 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.683144 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.683147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.683149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.683371 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:32.683471 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:32.684134 (+ 663us) server_negotiation.cc:299] Negotiation successful
0811 22:20:32.684273 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:32.685190 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.672898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36265 (local address 127.29.3.66:57847)
0811 22:20:32.673349 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:32.673393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:32.674692 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:32.675238 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:32.675245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:32.675595 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:32.676075 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:32.676089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.676888 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.676890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:32.677758 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:32.677767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.678317 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.678322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:32.678496 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.679065 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:32.679081 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:32.680540 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:32.682391 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:32.682396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:32.682408 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:32.682656 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:32.682935 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:32.682937 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:32.682938 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:32.683059 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:32.683572 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:32.683580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:32.683861 (+ 281us) client_negotiation.cc:769] Sending connection context
0811 22:20:32.684054 (+ 193us) client_negotiation.cc:240] Negotiation successful
0811 22:20:32.684325 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260811 22:20:32.686394 3604 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36265
I20260811 22:20:32.686679 3604 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:32.687317 3604 heartbeater.cc:511] Master 127.29.3.126:36265 requested a full tablet report, sending...
I20260811 22:20:32.688539 2997 ts_manager.cc:194] Registered new tserver with Master: d9b5faeca9c84d8d8b1ad8d995731628 (127.29.3.66:44275)
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:20:32.689134 2997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:57847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:32.709481 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36265
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:32.825953 3617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:32.826205 3617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:32.826247 3617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:32.830181 3617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:32.830236 3617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:32.830317 3617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:32.835532 3617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:32.836535 3617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:32.837474 3617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:32.846922 3636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.847080 3633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:32.847080 3639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:32.847236 3617 server_base.cc:1034] running on GCE node
I20260811 22:20:32.847631 3617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:32.848253 3617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:32.849406 3617 hybrid_clock.cc:648] HybridClock initialized: now 1786486832849385 us; error 35 us; skew 500 ppm
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:32.852082 3617 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:32.852967 3617 webserver.cc:460] Webserver started at http://127.29.3.67:34113/ using document root <none> and password file <none>
I20260811 22:20:32.853489 3617 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:32.853545 3617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:32.853718 3617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:32.855806 3617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "215d8110104a4fdea96aba3428e0a3b6"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "bc409fa2ac5ca1a8b9ce931efa0b9c13"
server_key_iv: "39d12c10cf737d420086077f8a57bfc3"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.856376 3617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "215d8110104a4fdea96aba3428e0a3b6"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "bc409fa2ac5ca1a8b9ce931efa0b9c13"
server_key_iv: "39d12c10cf737d420086077f8a57bfc3"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.860204 3617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:32.862751 3652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.864019 3617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:32.864135 3617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "215d8110104a4fdea96aba3428e0a3b6"
format_stamp: "Formatted at 2026-08-11 22:20:32 on dist-test-slave-4tdj"
server_key: "bc409fa2ac5ca1a8b9ce931efa0b9c13"
server_key_iv: "39d12c10cf737d420086077f8a57bfc3"
server_key_version: "encryptionkey@0"
I20260811 22:20:32.864238 3617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:32.876322 3617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:32.879839 3617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:32.880002 3617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:32.880555 3617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:32.881402 3617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:32.881450 3617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.881490 3617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:32.881510 3617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:32.890254 3617 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:43339
I20260811 22:20:32.890280 3850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:43339 every 8 connection(s)
I20260811 22:20:32.891463 3617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:32.896881 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3617
I20260811 22:20:32.896999 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:32.897277 29709 external_mini_cluster.cc:1428] Setting key 966ab58886768b8293e4b934d021b639
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:32.905018 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.893300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:42649 (local address 127.29.3.126:36265)
0811 22:20:32.893442 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:32.893445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:32.894269 (+ 824us) server_negotiation.cc:407] Connection header received
0811 22:20:32.895649 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:32.895651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:32.895701 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:32.895777 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:32.896938 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.897552 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.898842 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.899149 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.899248 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.901701 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:32.901704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:32.901706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:32.901745 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:32.903114 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.903645 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.903648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.903649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.903735 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.904032 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.904034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.904035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.904210 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:32.904284 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:32.904789 (+ 505us) server_negotiation.cc:299] Negotiation successful
0811 22:20:32.904895 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:20:32.905966 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.893646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36265 (local address 127.29.3.67:42649)
0811 22:20:32.894110 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:32.894153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:32.895451 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:32.895929 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:32.895936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:32.896304 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:32.896780 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:32.896793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.897698 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.897701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:32.898737 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:32.898748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.899303 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.899307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:32.899490 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.900031 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:32.900047 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:32.901534 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:32.903250 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:32.903255 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:32.903270 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:32.903534 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:32.903816 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:32.903819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:32.903821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:32.903963 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:32.904286 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:32.904291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:32.904536 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 22:20:32.904756 (+ 220us) client_negotiation.cc:240] Negotiation successful
0811 22:20:32.905024 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260811 22:20:32.907166 3851 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36265
I20260811 22:20:32.907402 3851 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:32.907910 3851 heartbeater.cc:511] Master 127.29.3.126:36265 requested a full tablet report, sending...
I20260811 22:20:32.908908 2997 ts_manager.cc:194] Registered new tserver with Master: 215d8110104a4fdea96aba3428e0a3b6 (127.29.3.67:43339)
I20260811 22:20:32.909468 2997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:42649
I20260811 22:20:32.911198 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 22:20:32 dist-test-slave-4tdj krb5kdc[2920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486832, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:32.937247 3062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:32.929272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.29.3.126:36265)
0811 22:20:32.929416 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:32.929420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:32.929558 (+ 138us) server_negotiation.cc:407] Connection header received
0811 22:20:32.929683 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:32.929686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:32.929731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:32.929809 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:32.930545 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.931075 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.931882 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:32.932084 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:32.932165 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:32.934481 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:32.934484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:32.934486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:32.934516 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:32.935729 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.936102 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.936104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.936105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.936179 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:32.936423 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:32.936427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:32.936428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:32.936698 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:32.936820 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:32.936996 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 22:20:32.937104 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260811 22:20:32.937674 2997 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39660
I20260811 22:20:32.938536 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3108
I20260811 22:20:32.951144 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3374
I20260811 22:20:32.961520 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3617
I20260811 22:20:33.008662 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2943
2026-08-11T22:20:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2375 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:33 dist-test-slave-4tdj krb5kdc[3876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:33 dist-test-slave-4tdj krb5kdc[3876](info): set up 2 sockets
Aug 11 22:20:33 dist-test-slave-4tdj krb5kdc[3876](info): commencing operation
krb5kdc: starting...
W20260811 22:20:34.201746 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:34 dist-test-slave-4tdj krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:34.326936 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42337
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:42337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:34.437888 3900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:34.438143 3900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:34.438199 3900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:34.442166 3900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:34.442231 3900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:34.442257 3900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:34.442278 3900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:34.442332 3900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:34.447631 3900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:42337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:34.448642 3900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:34.449651 3900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:34.457877 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:34.457872 3910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:34.457947 3900 server_base.cc:1034] running on GCE node
W20260811 22:20:34.457873 3911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:34.458719 3900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:34.459812 3900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:34.460983 3900 hybrid_clock.cc:648] HybridClock initialized: now 1786486834460966 us; error 32 us; skew 500 ppm
Aug 11 22:20:34 dist-test-slave-4tdj krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:34.463786 3900 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:34.464645 3900 webserver.cc:460] Webserver started at http://127.29.3.126:38431/ using document root <none> and password file <none>
I20260811 22:20:34.465170 3900 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:34.465248 3900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:34.465428 3900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:34.467475 3900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "2594aeb49b5a4986bfe73eaceac695ab"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "fb1f6111d544f836e4e94dee430d58ce"
server_key_iv: "54c40408101f7fff0007b24f5ed0dc7a"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.467979 3900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "2594aeb49b5a4986bfe73eaceac695ab"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "fb1f6111d544f836e4e94dee430d58ce"
server_key_iv: "54c40408101f7fff0007b24f5ed0dc7a"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.471639 3900 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:34.473949 3928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:34.475206 3900 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:34.475334 3900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "2594aeb49b5a4986bfe73eaceac695ab"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "fb1f6111d544f836e4e94dee430d58ce"
server_key_iv: "54c40408101f7fff0007b24f5ed0dc7a"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.475431 3900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:34.496502 3900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:34.499332 3900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:34.499522 3900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:34.506402 3900 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:42337
I20260811 22:20:34.506434 4028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:42337 every 8 connection(s)
I20260811 22:20:34.507347 3900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:34.510054 4029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:34.512892 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3900
I20260811 22:20:34.513016 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:34.513320 29709 external_mini_cluster.cc:1428] Setting key d1354b3bff6ed21ccec367c4692772e4
I20260811 22:20:34.516146 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab: Bootstrap starting.
I20260811 22:20:34.518273 4029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:34.519032 4029 log.cc:824] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:34.520861 4029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab: No bootstrap required, opened a new log
Aug 11 22:20:34 dist-test-slave-4tdj krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:34.524513 4029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2594aeb49b5a4986bfe73eaceac695ab" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42337 } }
I20260811 22:20:34.524767 4029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:34.524827 4029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2594aeb49b5a4986bfe73eaceac695ab, State: Initialized, Role: FOLLOWER
I20260811 22:20:34.525327 4029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2594aeb49b5a4986bfe73eaceac695ab" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42337 } }
I20260811 22:20:34.525496 4029 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:34.525568 4029 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:34.525681 4029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:34.526643 4029 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2594aeb49b5a4986bfe73eaceac695ab" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42337 } }
I20260811 22:20:34.526968 4029 leader_election.cc:302] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2594aeb49b5a4986bfe73eaceac695ab; no voters:
I20260811 22:20:34.527068 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:34.514774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.29.3.126:42337)
0811 22:20:34.515225 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:34.515241 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:34.515277 (+ 36us) server_negotiation.cc:407] Connection header received
0811 22:20:34.515904 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:34.515922 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:34.516252 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:34.516577 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:34.517374 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:34.518407 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:34.519222 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:34.519872 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:34.520023 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:34.522410 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:34.522415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:34.522427 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:34.522455 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:34.524056 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:34.524474 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:34.524480 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:34.524487 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:34.524618 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:34.524919 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:34.524923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:34.524924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:34.525463 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:34.525687 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:34.525971 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 22:20:34.526254 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260811 22:20:34.527276 4029 leader_election.cc:288] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:34.527627 4039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:34.527825 4039 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [term 1 LEADER]: Becoming Leader. State: Replica: 2594aeb49b5a4986bfe73eaceac695ab, State: Running, Role: LEADER
I20260811 22:20:34.528131 4039 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2594aeb49b5a4986bfe73eaceac695ab" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42337 } }
I20260811 22:20:34.528481 4029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:34.529856 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2594aeb49b5a4986bfe73eaceac695ab. Latest consensus state: current_term: 1 leader_uuid: "2594aeb49b5a4986bfe73eaceac695ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2594aeb49b5a4986bfe73eaceac695ab" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42337 } } }
I20260811 22:20:34.529990 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:34.530315 4041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2594aeb49b5a4986bfe73eaceac695ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2594aeb49b5a4986bfe73eaceac695ab" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42337 } } }
I20260811 22:20:34.530400 4057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:34.530418 4041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:34.534088 4057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:34.539008 4057 catalog_manager.cc:1347] Generated new cluster ID: e8e86263d5a44c56a137367b7277da38
I20260811 22:20:34.539057 4057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:34.548215 4057 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:34.549212 4057 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:34.567914 4057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2594aeb49b5a4986bfe73eaceac695ab: Generated new TSK 0
I20260811 22:20:34.568681 4057 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:34.597741 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42337
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:34.711315 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:34.711575 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:34.711615 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:34.715651 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:34.715726 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:34.715787 4068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:34.721095 4068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:34.722195 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:34.723284 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:34.732915 4079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:34.732908 4082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:34.732908 4075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:34.733026 4068 server_base.cc:1034] running on GCE node
I20260811 22:20:34.733551 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:34.734212 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:34.735371 4068 hybrid_clock.cc:648] HybridClock initialized: now 1786486834735352 us; error 30 us; skew 500 ppm
Aug 11 22:20:34 dist-test-slave-4tdj krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:34.738071 4068 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:34.738988 4068 webserver.cc:460] Webserver started at http://127.29.3.65:33691/ using document root <none> and password file <none>
I20260811 22:20:34.739534 4068 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:34.739604 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:34.739820 4068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:34.741915 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "89b5beb41b3646b1b45f50c19ceb49b8"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "5df4b031179102eac18dbdd969193168"
server_key_iv: "41426a5e147780bf856773141739556c"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.742511 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "89b5beb41b3646b1b45f50c19ceb49b8"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "5df4b031179102eac18dbdd969193168"
server_key_iv: "41426a5e147780bf856773141739556c"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.746280 4068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:20:34.748685 4101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:34.749866 4068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:20:34.749976 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "89b5beb41b3646b1b45f50c19ceb49b8"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "5df4b031179102eac18dbdd969193168"
server_key_iv: "41426a5e147780bf856773141739556c"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.750072 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:34.779649 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:34.782963 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:34.783165 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:34.783795 4068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:34.784667 4068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:34.784718 4068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:34.784773 4068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:34.784803 4068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:34.793210 4068 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:42337
I20260811 22:20:34.793241 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:42337 every 8 connection(s)
I20260811 22:20:34.794090 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:34.794548 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4068
I20260811 22:20:34.794677 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:34.795010 29709 external_mini_cluster.cc:1428] Setting key 77de9a1b3dbb28c0eba797f343331b42
Aug 11 22:20:34 dist-test-slave-4tdj krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:20:34.808846 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:34.796066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:39193 (local address 127.29.3.126:42337)
0811 22:20:34.796187 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:34.796190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:34.797280 (+ 1090us) server_negotiation.cc:407] Connection header received
0811 22:20:34.799062 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:34.799065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:34.799125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:34.799217 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:34.800407 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:34.801160 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:34.802406 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:34.802779 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:34.802894 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:34.805607 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:34.805611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:34.805613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:34.805639 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:34.807045 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:34.807504 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:34.807506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:34.807508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:34.807578 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:34.807836 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:34.807838 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:34.807840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:34.808026 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:34.808129 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:34.808581 (+ 452us) server_negotiation.cc:299] Negotiation successful
0811 22:20:34.808708 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:20:34.809737 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:34.796559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42337 (local address 127.29.3.65:39193)
0811 22:20:34.797094 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:34.797154 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:34.798826 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:34.799377 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:34.799383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:34.799751 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:34.800245 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:34.800261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:34.801275 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:34.801278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:34.802240 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:34.802251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:34.802928 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:34.802930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:34.803081 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:34.803668 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:34.803685 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:34.805427 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:34.807152 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:34.807158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:34.807171 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:34.807418 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:34.807643 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:34.807645 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:34.807647 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:34.807772 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:34.808142 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:34.808147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:34.808385 (+ 238us) client_negotiation.cc:769] Sending connection context
0811 22:20:34.808542 (+ 157us) client_negotiation.cc:240] Negotiation successful
0811 22:20:34.808881 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":120,"threads_started":1}
I20260811 22:20:34.811044 4307 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42337
I20260811 22:20:34.811313 4307 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:34.811918 4307 heartbeater.cc:511] Master 127.29.3.126:42337 requested a full tablet report, sending...
I20260811 22:20:34.813513 3962 ts_manager.cc:194] Registered new tserver with Master: 89b5beb41b3646b1b45f50c19ceb49b8 (127.29.3.65:42337)
I20260811 22:20:34.814697 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:39193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:34.825912 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42337
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:34.937711 4316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:34.937945 4316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:34.938004 4316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:34.941968 4316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:34.942031 4316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:34.942111 4316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:34.947228 4316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:34.948225 4316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:34.949203 4316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:34.958539 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:34.958730 4316 server_base.cc:1034] running on GCE node
W20260811 22:20:34.958523 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:34.958523 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:34.959232 4316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:34.959846 4316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:34.961009 4316 hybrid_clock.cc:648] HybridClock initialized: now 1786486834960987 us; error 34 us; skew 500 ppm
Aug 11 22:20:34 dist-test-slave-4tdj krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:34.963732 4316 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:34.964599 4316 webserver.cc:460] Webserver started at http://127.29.3.66:33133/ using document root <none> and password file <none>
I20260811 22:20:34.965144 4316 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:34.965234 4316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:34.965457 4316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:34.967585 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "51cee3a172474eadbd39e464e06e8a72"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "56491909bdc2fc877d6c23574af979cf"
server_key_iv: "b9acd0810e9403f9bc4604302d1eee00"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.968151 4316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "51cee3a172474eadbd39e464e06e8a72"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "56491909bdc2fc877d6c23574af979cf"
server_key_iv: "b9acd0810e9403f9bc4604302d1eee00"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.972126 4316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:34.974659 4340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:34.975910 4316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:34.976037 4316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "51cee3a172474eadbd39e464e06e8a72"
format_stamp: "Formatted at 2026-08-11 22:20:34 on dist-test-slave-4tdj"
server_key: "56491909bdc2fc877d6c23574af979cf"
server_key_iv: "b9acd0810e9403f9bc4604302d1eee00"
server_key_version: "encryptionkey@0"
I20260811 22:20:34.976140 4316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:34.997953 4316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:35.000854 4316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:35.001040 4316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:35.001670 4316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:35.002645 4316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:35.002693 4316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:35.002733 4316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:35.002753 4316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:35.012154 4316 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:43611
I20260811 22:20:35.012214 4546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:43611 every 8 connection(s)
I20260811 22:20:35.013092 4316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:35.022615 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4316
I20260811 22:20:35.022751 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:35.023068 29709 external_mini_cluster.cc:1428] Setting key 7c63332397e8d6ad5746097d60d353e5
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486834, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:35.028570 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:35.015039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:37179 (local address 127.29.3.126:42337)
0811 22:20:35.015202 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:35.015206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:35.016016 (+ 810us) server_negotiation.cc:407] Connection header received
0811 22:20:35.017435 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:35.017438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:35.017487 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:35.017561 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:35.018811 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.019337 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.021045 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.021380 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.021475 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:35.024567 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:35.024570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:35.024572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:35.024598 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:35.026154 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.026797 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.026801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.026803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.026887 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.027282 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.027286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.027288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.027508 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:35.027589 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:35.028277 (+ 688us) server_negotiation.cc:299] Negotiation successful
0811 22:20:35.028411 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 22:20:35.029662 4552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:35.015369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42337 (local address 127.29.3.66:37179)
0811 22:20:35.015850 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:35.015896 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:35.017223 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:35.017727 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:35.017733 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:35.018113 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:35.018668 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:35.018684 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.019464 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.019468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:35.020878 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:35.020892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.021544 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.021549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:35.021761 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:35.022538 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:35.022559 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:35.024374 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:35.026331 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:35.026338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:35.026355 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:35.026682 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:35.026991 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:35.026994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:35.026997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:35.027179 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:35.027659 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:35.027666 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:35.028020 (+ 354us) client_negotiation.cc:769] Sending connection context
0811 22:20:35.028241 (+ 221us) client_negotiation.cc:240] Negotiation successful
0811 22:20:35.028577 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":115,"threads_started":1}
I20260811 22:20:35.030920 4547 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42337
I20260811 22:20:35.031180 4547 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:35.031688 4547 heartbeater.cc:511] Master 127.29.3.126:42337 requested a full tablet report, sending...
I20260811 22:20:35.032790 3962 ts_manager.cc:194] Registered new tserver with Master: 51cee3a172474eadbd39e464e06e8a72 (127.29.3.66:43611)
I20260811 22:20:35.033382 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:37179
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:35.055459 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42337
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:35.169143 4558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:35.169422 4558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:35.169481 4558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:35.173543 4558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:35.173617 4558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:35.173695 4558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:35.179081 4558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42337
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:35.180215 4558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:35.181262 4558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:35.191069 4570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:35.191087 4569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:35.191177 4573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:35.191339 4558 server_base.cc:1034] running on GCE node
I20260811 22:20:35.191774 4558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:35.192402 4558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:35.193554 4558 hybrid_clock.cc:648] HybridClock initialized: now 1786486835193528 us; error 40 us; skew 500 ppm
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486835, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:35.196298 4558 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:35.197193 4558 webserver.cc:460] Webserver started at http://127.29.3.67:42767/ using document root <none> and password file <none>
I20260811 22:20:35.197726 4558 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:35.197785 4558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:35.197956 4558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:35.200037 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "03fca83e51344bacaa01df53097fdeb0"
format_stamp: "Formatted at 2026-08-11 22:20:35 on dist-test-slave-4tdj"
server_key: "8a3384b60e9fbfc7359a82a20ea57912"
server_key_iv: "f220cb82bec4dd7e94b92cc853564f30"
server_key_version: "encryptionkey@0"
I20260811 22:20:35.200546 4558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "03fca83e51344bacaa01df53097fdeb0"
format_stamp: "Formatted at 2026-08-11 22:20:35 on dist-test-slave-4tdj"
server_key: "8a3384b60e9fbfc7359a82a20ea57912"
server_key_iv: "f220cb82bec4dd7e94b92cc853564f30"
server_key_version: "encryptionkey@0"
I20260811 22:20:35.204301 4558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 22:20:35.206879 4587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:35.208139 4558 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:35.208245 4558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "03fca83e51344bacaa01df53097fdeb0"
format_stamp: "Formatted at 2026-08-11 22:20:35 on dist-test-slave-4tdj"
server_key: "8a3384b60e9fbfc7359a82a20ea57912"
server_key_iv: "f220cb82bec4dd7e94b92cc853564f30"
server_key_version: "encryptionkey@0"
I20260811 22:20:35.208356 4558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:35.220599 4558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:35.223790 4558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:35.223965 4558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:35.224560 4558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:35.225412 4558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:35.225461 4558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:35.225504 4558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:35.225533 4558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:35.234555 4558 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:46715
I20260811 22:20:35.234611 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:46715 every 8 connection(s)
I20260811 22:20:35.235432 4558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:35.242352 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4558
I20260811 22:20:35.242497 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:35.242846 29709 external_mini_cluster.cc:1428] Setting key a019ae9c24b595ed1fb0a888248f5338
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486835, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:35.248721 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:35.237183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:36229 (local address 127.29.3.126:42337)
0811 22:20:35.237324 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:35.237328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:35.238219 (+ 891us) server_negotiation.cc:407] Connection header received
0811 22:20:35.239614 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:35.239618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:35.239668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:35.239743 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:35.240859 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.241378 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.242837 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.243143 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.243269 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:35.245817 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:35.245820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:35.245822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:35.245848 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:35.247032 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.247464 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.247467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.247468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.247537 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.247814 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.247816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.247817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.247983 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:35.248053 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:35.248499 (+ 446us) server_negotiation.cc:299] Negotiation successful
0811 22:20:35.248598 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:35.249548 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:35.237577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42337 (local address 127.29.3.67:36229)
0811 22:20:35.238065 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:35.238108 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:35.239402 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:35.239893 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:35.239900 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:35.240244 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:35.240743 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:35.240758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.241465 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.241467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:35.242709 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:35.242721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.243322 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.243327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:35.243526 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:35.244138 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:35.244159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:35.245664 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:35.247131 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:35.247136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:35.247149 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:35.247388 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:35.247619 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:35.247621 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:35.247623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:35.247749 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:35.248054 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:35.248058 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:35.248303 (+ 245us) client_negotiation.cc:769] Sending connection context
0811 22:20:35.248453 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 22:20:35.248720 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260811 22:20:35.250649 4786 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42337
I20260811 22:20:35.250872 4786 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:35.251322 4786 heartbeater.cc:511] Master 127.29.3.126:42337 requested a full tablet report, sending...
I20260811 22:20:35.252300 3962 ts_manager.cc:194] Registered new tserver with Master: 03fca83e51344bacaa01df53097fdeb0 (127.29.3.67:46715)
I20260811 22:20:35.252822 3962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:36229
I20260811 22:20:35.256408 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[3876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:35.283034 4796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:35.273368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50434 (local address 127.29.3.65:42337)
0811 22:20:35.273722 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:35.273740 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:35.273764 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:20:35.273817 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:35.273822 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:35.274026 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:35.274159 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:35.274917 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.275718 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.276581 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.276953 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.277056 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:35.279447 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:35.279457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:35.279471 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:35.279527 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:35.281230 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.281656 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.281661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.281662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.281754 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.281958 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.281960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.281961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.282213 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:35.282438 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:35.282655 (+ 217us) server_negotiation.cc:299] Negotiation successful
0811 22:20:35.282822 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":92,"threads_started":1}
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[3876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:35.293635 4033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:35.285944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.29.3.126:42337)
0811 22:20:35.286104 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:35.286107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:35.286122 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:35.286256 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:35.286259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:35.286349 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:35.286419 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:35.286958 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.287478 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.288269 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:35.288505 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:35.288606 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:35.290885 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:35.290888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:35.290890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:35.290916 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:35.292024 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.292420 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.292423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.292424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.292502 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:35.292770 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:35.292773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:35.292775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:35.292996 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:35.293121 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:35.293363 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:20:35.293480 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260811 22:20:35.294059 3962 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37084
I20260811 22:20:35.294670 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4068
I20260811 22:20:35.329671 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4316
I20260811 22:20:35.339421 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4558
I20260811 22:20:35.357597 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3900
2026-08-11T22:20:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2355 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[4803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[4803](info): set up 2 sockets
Aug 11 22:20:35 dist-test-slave-4tdj krb5kdc[4803](info): commencing operation
krb5kdc: starting...
W20260811 22:20:36.552807 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:36 dist-test-slave-4tdj krb5kdc[4803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:36.676959 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39173
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:36.788625 4818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:36.788936 4818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:36.789004 4818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:36.793383 4818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:36.793468 4818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:36.793500 4818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:36.793524 4818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:36.793543 4818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:36.799086 4818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:36.800231 4818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:36.801277 4818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:36.809620 4830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:36.809677 4826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:36.809636 4828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:36.809846 4818 server_base.cc:1034] running on GCE node
I20260811 22:20:36.810539 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:36.811655 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:36.812829 4818 hybrid_clock.cc:648] HybridClock initialized: now 1786486836812809 us; error 32 us; skew 500 ppm
Aug 11 22:20:36 dist-test-slave-4tdj krb5kdc[4803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486836, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:36.815550 4818 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:36.816217 4818 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:36.816279 4818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:36.816442 4818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:36.818447 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b"
format_stamp: "Formatted at 2026-08-11 22:20:36 on dist-test-slave-4tdj"
server_key: "33cfe9fcfcbc8cda8e460ebc056fc361"
server_key_iv: "3148871c483bb622f865156e14264727"
server_key_version: "encryptionkey@0"
I20260811 22:20:36.818991 4818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b"
format_stamp: "Formatted at 2026-08-11 22:20:36 on dist-test-slave-4tdj"
server_key: "33cfe9fcfcbc8cda8e460ebc056fc361"
server_key_iv: "3148871c483bb622f865156e14264727"
server_key_version: "encryptionkey@0"
I20260811 22:20:36.822711 4818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:36.825156 4841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:36.826432 4818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:36.826565 4818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b"
format_stamp: "Formatted at 2026-08-11 22:20:36 on dist-test-slave-4tdj"
server_key: "33cfe9fcfcbc8cda8e460ebc056fc361"
server_key_iv: "3148871c483bb622f865156e14264727"
server_key_version: "encryptionkey@0"
I20260811 22:20:36.826664 4818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:36.839784 4818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:36.842844 4818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:36.843017 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:36.849553 4818 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:39173
I20260811 22:20:36.849781 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:39173 every 8 connection(s)
I20260811 22:20:36.850224 4818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:36.852957 4943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:36.853607 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4818
I20260811 22:20:36.853715 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:36.853963 29709 external_mini_cluster.cc:1428] Setting key 19e5c3d6d696a6f0a46c24962f45e94b
I20260811 22:20:36.859496 4943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b: Bootstrap starting.
I20260811 22:20:36.862013 4943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:36.862833 4943 log.cc:824] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:20:36 dist-test-slave-4tdj krb5kdc[4803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:36.865033 4943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b: No bootstrap required, opened a new log
I20260811 22:20:36.868412 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:36.855347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.29.3.126:39173)
0811 22:20:36.855793 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:36.855813 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:36.855857 (+ 44us) server_negotiation.cc:407] Connection header received
0811 22:20:36.856650 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:36.856675 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:36.857102 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:36.857453 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:36.858113 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:36.859550 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:36.860399 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:36.861077 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:36.861230 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:36.863665 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:36.863671 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:36.863687 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:36.863724 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:36.865482 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:36.865883 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:36.865889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:36.865895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:36.866033 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:36.866254 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:36.866257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:36.866258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:36.866780 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:36.866966 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:36.867172 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 22:20:36.867486 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":107,"threads_started":1}
I20260811 22:20:36.869422 4943 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39173 } }
I20260811 22:20:36.869655 4943 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:36.869705 4943 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a7d778d4dc4fa3bbcf255bbe28289b, State: Initialized, Role: FOLLOWER
I20260811 22:20:36.870146 4943 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39173 } }
I20260811 22:20:36.870281 4943 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:36.870391 4943 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:36.870498 4943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:36.871363 4943 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39173 } }
I20260811 22:20:36.871681 4943 leader_election.cc:302] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0a7d778d4dc4fa3bbcf255bbe28289b; no voters:
I20260811 22:20:36.871968 4943 leader_election.cc:288] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:36.872066 4951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:36.872289 4951 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [term 1 LEADER]: Becoming Leader. State: Replica: b0a7d778d4dc4fa3bbcf255bbe28289b, State: Running, Role: LEADER
I20260811 22:20:36.872601 4951 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39173 } }
I20260811 22:20:36.873076 4943 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:36.874370 4952 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39173 } } }
I20260811 22:20:36.874269 4955 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a7d778d4dc4fa3bbcf255bbe28289b. Latest consensus state: current_term: 1 leader_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a7d778d4dc4fa3bbcf255bbe28289b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39173 } } }
I20260811 22:20:36.874554 4952 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:36.874568 4955 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:36.875408 4970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:36.878430 4970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:36.883872 4970 catalog_manager.cc:1347] Generated new cluster ID: fac6b711bd48423da187dbaafef9469b
I20260811 22:20:36.883949 4970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:36.910426 4970 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:36.911511 4970 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:36.916775 4970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b: Generated new TSK 0
I20260811 22:20:36.917533 4970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:36.954651 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39173
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:37.072404 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:37.072700 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:37.072747 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:37.076951 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:37.077020 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:37.077087 4999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:37.082422 4999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:37.083524 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:37.084613 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:37.094173 5016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:37.094244 4999 server_base.cc:1034] running on GCE node
W20260811 22:20:37.094153 5010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:37.094169 5009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:37.095453 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:37.096077 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:37.097239 4999 hybrid_clock.cc:648] HybridClock initialized: now 1786486837097212 us; error 43 us; skew 500 ppm
Aug 11 22:20:37 dist-test-slave-4tdj krb5kdc[4803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486837, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:37.104694 4999 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:37.105535 4999 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:37.105609 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:37.105814 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:37.107980 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "3e9da459ce8e46ca98cfd45c509c9031"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "dcdf2d7201f3e06316edcc1519e1777a"
server_key_iv: "faf5790df0af74a60f072475508ea7cb"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.108549 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "3e9da459ce8e46ca98cfd45c509c9031"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "dcdf2d7201f3e06316edcc1519e1777a"
server_key_iv: "faf5790df0af74a60f072475508ea7cb"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.112629 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:37.115271 5028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.116557 4999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260811 22:20:37.116683 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "3e9da459ce8e46ca98cfd45c509c9031"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "dcdf2d7201f3e06316edcc1519e1777a"
server_key_iv: "faf5790df0af74a60f072475508ea7cb"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.116786 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:37.127084 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:37.130182 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:37.130396 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:37.131006 4999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:37.131930 4999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:37.131981 4999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.132037 4999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:37.132068 4999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.140633 4999 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:35023
I20260811 22:20:37.140679 5239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:35023 every 8 connection(s)
I20260811 22:20:37.141355 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:37.142112 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4999
I20260811 22:20:37.142235 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:37.142535 29709 external_mini_cluster.cc:1428] Setting key f6f507582bd9ca493cc7e63f33cb5d50
Aug 11 22:20:37 dist-test-slave-4tdj krb5kdc[4803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486837, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:20:37.155844 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.143199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:50007 (local address 127.29.3.126:39173)
0811 22:20:37.143356 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.143361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.144338 (+ 977us) server_negotiation.cc:407] Connection header received
0811 22:20:37.145709 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.145713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.145770 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.145862 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:37.147449 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.147983 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.149127 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.149363 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.149453 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.152124 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:37.152128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:37.152130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:37.152165 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:37.153616 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.154133 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.154136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.154137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.154228 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.154630 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.154634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.154636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.154859 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:37.154953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.155522 (+ 569us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.155670 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:20:37.156692 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.143615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39173 (local address 127.29.3.65:50007)
0811 22:20:37.144159 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.144223 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.145489 (+ 1266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.146072 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.146082 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.146617 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:37.147319 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.147337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.148116 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.148119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.148998 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.149008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.149518 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.149522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.149689 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.150251 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:37.150268 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:37.151938 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:37.153737 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:37.153744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:37.153762 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:37.154025 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:37.154343 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:37.154346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:37.154348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:37.154510 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:37.154965 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:37.154973 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:37.155266 (+ 293us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.155440 (+ 174us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.155806 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":152,"threads_started":1}
I20260811 22:20:37.158026 5240 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39173
I20260811 22:20:37.158330 5240 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:37.158910 5240 heartbeater.cc:511] Master 127.29.3.126:39173 requested a full tablet report, sending...
I20260811 22:20:37.160703 4873 ts_manager.cc:194] Registered new tserver with Master: 3e9da459ce8e46ca98cfd45c509c9031 (127.29.3.65:35023)
I20260811 22:20:37.162381 4873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:50007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:37.174072 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39173
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:37.288050 5246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:37.288311 5246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:37.288355 5246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:37.292366 5246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:37.292423 5246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:37.292482 5246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:37.297681 5246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:37.298761 5246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:37.299751 5246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:37.309302 5259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:37.309360 5257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:37.309273 5255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:37.309780 5246 server_base.cc:1034] running on GCE node
I20260811 22:20:37.310173 5246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:37.310853 5246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:37.312009 5246 hybrid_clock.cc:648] HybridClock initialized: now 1786486837311989 us; error 32 us; skew 500 ppm
Aug 11 22:20:37 dist-test-slave-4tdj krb5kdc[4803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486837, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:37.314759 5246 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:37.315474 5246 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:37.315538 5246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:37.315706 5246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:37.317739 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "cffe2acce7864676947d85f51fca87b8"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "b7cb646c91d3c3f3019769f77be1ae00"
server_key_iv: "6dc20a4dc4078be0822c65037514cb5e"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.318259 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "cffe2acce7864676947d85f51fca87b8"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "b7cb646c91d3c3f3019769f77be1ae00"
server_key_iv: "6dc20a4dc4078be0822c65037514cb5e"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.322242 5246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:20:37.324682 5273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.325928 5246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:37.326035 5246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "cffe2acce7864676947d85f51fca87b8"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "b7cb646c91d3c3f3019769f77be1ae00"
server_key_iv: "6dc20a4dc4078be0822c65037514cb5e"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.326143 5246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:37.338521 5246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:37.341133 5246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:37.341301 5246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:37.341881 5246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:37.342780 5246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:37.342829 5246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.342871 5246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:37.342900 5246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.351066 5246 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:33567
I20260811 22:20:37.351104 5472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:33567 every 8 connection(s)
I20260811 22:20:37.351630 5246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
Aug 11 22:20:37 dist-test-slave-4tdj krb5kdc[4803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486837, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:37.361277 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5246
I20260811 22:20:37.361385 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:37.361886 29709 external_mini_cluster.cc:1428] Setting key 9de14e46bbf9e9d92bbd43dd51cb842a
I20260811 22:20:37.364809 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.353293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:47949 (local address 127.29.3.126:39173)
0811 22:20:37.353433 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.353437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.354249 (+ 812us) server_negotiation.cc:407] Connection header received
0811 22:20:37.355625 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.355628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.355685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.355769 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:37.356931 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.357454 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.358637 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.358873 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.358965 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.361198 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:37.361201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:37.361202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:37.361230 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:37.362631 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.363288 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.363292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.363294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.363391 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.363763 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.363766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.363768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.363948 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:37.364038 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.364519 (+ 481us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.364643 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:37.365741 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.353649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39173 (local address 127.29.3.66:47949)
0811 22:20:37.354096 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.354138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.355407 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.355957 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.355964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.356318 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:37.356779 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.356792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.357578 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.357581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.358518 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.358529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.359018 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.359022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.359184 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.359674 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:37.359689 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:37.361035 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:37.362734 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:37.362740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:37.362753 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:37.363000 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:37.363492 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:37.363494 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:37.363496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:37.363650 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:37.364057 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:37.364063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:37.364294 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.364455 (+ 161us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.364752 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":128,"threads_started":1}
I20260811 22:20:37.366873 5473 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39173
I20260811 22:20:37.367089 5473 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:37.367560 5473 heartbeater.cc:511] Master 127.29.3.126:39173 requested a full tablet report, sending...
I20260811 22:20:37.368785 4873 ts_manager.cc:194] Registered new tserver with Master: cffe2acce7864676947d85f51fca87b8 (127.29.3.66:33567)
I20260811 22:20:37.369552 4873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:47949
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:37.390039 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39173
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:37.499912 5492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:37.500146 5492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:37.500201 5492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:37.504084 5492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:37.504138 5492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:37.504215 5492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:37.509248 5492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:37.510263 5492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:37.511243 5492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:37.520188 5503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:37.520213 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:37.520341 5492 server_base.cc:1034] running on GCE node
W20260811 22:20:37.520206 5509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:37.520814 5492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:37.521436 5492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:37.522588 5492 hybrid_clock.cc:648] HybridClock initialized: now 1786486837522568 us; error 30 us; skew 500 ppm
Aug 11 22:20:37 dist-test-slave-4tdj krb5kdc[4803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486837, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:37.525286 5492 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:37.525979 5492 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:37.526041 5492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:37.526226 5492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:37.528208 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "2ef19082422a428bbb516d98c1e97188"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "bb42ffedd1d8d60c908c8aab7b8d61c5"
server_key_iv: "cdb9dac4514fe5272ca6c077ba1bafc5"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.528738 5492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "2ef19082422a428bbb516d98c1e97188"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "bb42ffedd1d8d60c908c8aab7b8d61c5"
server_key_iv: "cdb9dac4514fe5272ca6c077ba1bafc5"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.532523 5492 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260811 22:20:37.534847 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.536022 5492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260811 22:20:37.536119 5492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "2ef19082422a428bbb516d98c1e97188"
format_stamp: "Formatted at 2026-08-11 22:20:37 on dist-test-slave-4tdj"
server_key: "bb42ffedd1d8d60c908c8aab7b8d61c5"
server_key_iv: "cdb9dac4514fe5272ca6c077ba1bafc5"
server_key_version: "encryptionkey@0"
I20260811 22:20:37.536222 5492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:37.565596 5492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:37.569784 5492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:37.569960 5492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:37.570708 5492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:37.571722 5492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:37.571776 5492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.571832 5492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:37.571861 5492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:37.580749 5492 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:45203
I20260811 22:20:37.580762 5706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:45203 every 8 connection(s)
I20260811 22:20:37.581349 5492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:37.587625 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5492
I20260811 22:20:37.587759 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:37.588021 29709 external_mini_cluster.cc:1428] Setting key 9168d5c7fbf2fc26baa6a08151a74bef
Aug 11 22:20:37 dist-test-slave-4tdj krb5kdc[4803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486837, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:37.594825 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.583147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:57431 (local address 127.29.3.126:39173)
0811 22:20:37.583297 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.583301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.584064 (+ 763us) server_negotiation.cc:407] Connection header received
0811 22:20:37.585426 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.585429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.585488 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.585559 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:37.586822 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.587588 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.589032 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.589330 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.589436 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.591910 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:37.591913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:37.591915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:37.591940 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:37.593047 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.593527 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.593529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.593531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.593597 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.593860 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.593862 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.593863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.594028 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:37.594098 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.594592 (+ 494us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.594696 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:20:37.595644 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.583468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39173 (local address 127.29.3.67:57431)
0811 22:20:37.583912 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.583955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.585179 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.585759 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.585766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.586123 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:37.586670 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.586685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.587728 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.587732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.588919 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.588931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.589489 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.589494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.589689 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.590355 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:37.590375 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:37.591765 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:37.593144 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:37.593149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:37.593168 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:37.593451 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:37.593662 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:37.593665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:37.593667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:37.593792 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:37.594103 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:37.594107 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:37.594383 (+ 276us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.594544 (+ 161us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.594791 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260811 22:20:37.596748 5707 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39173
I20260811 22:20:37.596977 5707 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:37.597419 5707 heartbeater.cc:511] Master 127.29.3.126:39173 requested a full tablet report, sending...
I20260811 22:20:37.598407 4873 ts_manager.cc:194] Registered new tserver with Master: 2ef19082422a428bbb516d98c1e97188 (127.29.3.67:45203)
I20260811 22:20:37.598973 4873 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:57431
I20260811 22:20:37.601791 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:20:37.610927 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.603686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.29.3.126:39173)
0811 22:20:37.603802 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.603807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.603980 (+ 173us) server_negotiation.cc:407] Connection header received
0811 22:20:37.604109 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.604112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.604154 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.604229 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:37.604943 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.605459 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.606318 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.606544 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.606638 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.607938 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:37.607942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:37.607945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:37.607969 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:37.609249 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.609642 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.609645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.609646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.609723 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:37.609975 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:37.609978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:37.609980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:37.610250 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:37.610440 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.610663 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.610776 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:20:37.613687 4873 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:37.616809 4873 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:37.631287 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.626225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.29.3.67:45203)
0811 22:20:37.626716 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.626727 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.626750 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:20:37.626923 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.626929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.627118 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.627253 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:37.627470 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.628235 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.629851 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.630575 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.630684 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.630791 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.631006 (+ 215us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.631123 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":197,"threads_started":1}
I20260811 22:20:37.631441 5736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.625567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35023 (local address 127.0.0.1:39466)
0811 22:20:37.626184 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.626197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.626285 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.627001 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.627003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.627019 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:37.627098 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.627103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.628426 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.628429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.629878 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.629886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.630930 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.630934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.631041 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.631066 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.631139 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.631208 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":21,"thread_start_us":115,"threads_started":1}
I20260811 22:20:37.631493 5730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.624838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:33567 (local address 127.0.0.1:35216)
0811 22:20:37.625160 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.625201 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.625340 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.625932 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.625937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.625961 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:37.626119 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.626127 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.627623 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.627626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.628819 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.628830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.631121 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.631124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.631217 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.631238 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.631309 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.631380 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260811 22:20:37.631634 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.625029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.29.3.66:33567)
0811 22:20:37.625427 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.625440 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.625463 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:20:37.625545 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.625550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.625764 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.625931 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:37.626257 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.627497 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.629090 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.629882 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.630030 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.630199 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.631347 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.631496 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":185,"threads_started":1}
I20260811 22:20:37.631263 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.625911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:45203 (local address 127.0.0.1:40350)
0811 22:20:37.626655 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.626668 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.626747 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.627239 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.627241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.627258 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:37.627354 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.627360 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.628368 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.628371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.629655 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.629665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.630686 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.630689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.630808 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.630855 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.630960 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.631094 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":104,"thread_start_us":77,"threads_started":1}
I20260811 22:20:37.632164 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.625735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.29.3.65:35023)
0811 22:20:37.626245 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.626251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.626273 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:20:37.626476 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.626482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.626685 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.626835 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:37.627222 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.628296 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.630000 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.630810 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.630921 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.631060 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.631895 (+ 835us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.631993 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":91,"threads_started":1}
I20260811 22:20:37.634693 5366 tablet_service.cc:1467] Processing CreateTablet for tablet d370954d216c4511a079e55ec47ec5cd (DEFAULT_TABLE table=test-table [id=c9d5bccd94244eeaa97588ff2838d423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:37.634768 5586 tablet_service.cc:1467] Processing CreateTablet for tablet d370954d216c4511a079e55ec47ec5cd (DEFAULT_TABLE table=test-table [id=c9d5bccd94244eeaa97588ff2838d423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:37.634768 5116 tablet_service.cc:1467] Processing CreateTablet for tablet d370954d216c4511a079e55ec47ec5cd (DEFAULT_TABLE table=test-table [id=c9d5bccd94244eeaa97588ff2838d423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:37.635882 5116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d370954d216c4511a079e55ec47ec5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:37.635885 5586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d370954d216c4511a079e55ec47ec5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:37.635883 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d370954d216c4511a079e55ec47ec5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:37.642558 5742 tablet_bootstrap.cc:492] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Bootstrap starting.
I20260811 22:20:37.643997 5743 tablet_bootstrap.cc:492] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: Bootstrap starting.
I20260811 22:20:37.645503 5742 tablet_bootstrap.cc:654] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:37.645741 5743 tablet_bootstrap.cc:654] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:37.646658 5742 log.cc:824] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:37.646672 5744 tablet_bootstrap.cc:492] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Bootstrap starting.
I20260811 22:20:37.646903 5743 log.cc:824] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:37.648732 5743 tablet_bootstrap.cc:492] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: No bootstrap required, opened a new log
I20260811 22:20:37.648751 5742 tablet_bootstrap.cc:492] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: No bootstrap required, opened a new log
I20260811 22:20:37.648782 5744 tablet_bootstrap.cc:654] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:37.648944 5742 ts_tablet_manager.cc:1397] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 22:20:37.648979 5743 ts_tablet_manager.cc:1397] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20260811 22:20:37.650115 5744 log.cc:824] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:37.652163 5744 tablet_bootstrap.cc:492] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: No bootstrap required, opened a new log
I20260811 22:20:37.652072 5743 raft_consensus.cc:348] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.652297 5743 raft_consensus.cc:374] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:37.652364 5744 ts_tablet_manager.cc:1397] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:20:37.652369 5743 raft_consensus.cc:729] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9da459ce8e46ca98cfd45c509c9031, State: Initialized, Role: FOLLOWER
I20260811 22:20:37.652578 5742 raft_consensus.cc:348] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.652832 5742 raft_consensus.cc:374] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:37.652812 5743 consensus_queue.cc:260] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.652937 5742 raft_consensus.cc:729] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffe2acce7864676947d85f51fca87b8, State: Initialized, Role: FOLLOWER
I20260811 22:20:37.653491 5240 heartbeater.cc:503] Master 127.29.3.126:39173 was elected leader, sending a full tablet report...
I20260811 22:20:37.653431 5742 consensus_queue.cc:260] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.653820 5743 ts_tablet_manager.cc:1428] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260811 22:20:37.654381 5742 ts_tablet_manager.cc:1428] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:20:37.654739 5473 heartbeater.cc:503] Master 127.29.3.126:39173 was elected leader, sending a full tablet report...
I20260811 22:20:37.655910 5744 raft_consensus.cc:348] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.656178 5744 raft_consensus.cc:374] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:37.656275 5744 raft_consensus.cc:729] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef19082422a428bbb516d98c1e97188, State: Initialized, Role: FOLLOWER
I20260811 22:20:37.656674 5744 consensus_queue.cc:260] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.657269 5707 heartbeater.cc:503] Master 127.29.3.126:39173 was elected leader, sending a full tablet report...
I20260811 22:20:37.657572 5744 ts_tablet_manager.cc:1428] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260811 22:20:37.832237 5708 tablet.cc:2369] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:37.852715 5479 tablet.cc:2369] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:37.865585 5749 raft_consensus.cc:482] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:37.865809 5749 raft_consensus.cc:504] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.867031 5749 leader_election.cc:288] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cffe2acce7864676947d85f51fca87b8 (127.29.3.66:33567), 2ef19082422a428bbb516d98c1e97188 (127.29.3.67:45203)
I20260811 22:20:37.870923 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.867443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:34825 (local address 127.29.3.66:33567)
0811 22:20:37.867612 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.867617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.867705 (+ 88us) server_negotiation.cc:407] Connection header received
0811 22:20:37.867887 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.867891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.867949 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.868029 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:37.868323 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.868836 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.869948 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.870398 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.870503 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.870578 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.870731 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.870801 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260811 22:20:37.870962 5755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.867300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:33567 (local address 127.29.3.65:34825)
0811 22:20:37.867611 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.867626 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.867752 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.868052 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.868054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.868117 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:37.868219 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.868225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.868942 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.868946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.869831 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.869840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.870510 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.870513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.870611 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.870635 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.870707 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.870787 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":44,"spinlock_wait_cycles":37632,"thread_start_us":117,"threads_started":1}
I20260811 22:20:37.871107 5741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.867843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:58103 (local address 127.29.3.67:45203)
0811 22:20:37.867970 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.867975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.868064 (+ 89us) server_negotiation.cc:407] Connection header received
0811 22:20:37.868223 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.868226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.868270 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.868334 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:37.868590 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.869065 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.870108 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.870565 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.870662 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.870723 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.870916 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.870983 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:20:37.871261 5756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.867689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:45203 (local address 127.29.3.65:58103)
0811 22:20:37.867968 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.867993 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:37.868125 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:37.868354 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:37.868356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:37.868371 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:37.868483 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.868489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.869172 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.869175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.869970 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:37.869987 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.870682 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.870686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:37.870803 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.870822 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:20:37.870887 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 22:20:37.870955 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":4,"spinlock_wait_cycles":37376,"thread_start_us":104,"threads_started":1}
I20260811 22:20:37.871713 5395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d370954d216c4511a079e55ec47ec5cd" candidate_uuid: "3e9da459ce8e46ca98cfd45c509c9031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffe2acce7864676947d85f51fca87b8" is_pre_election: true
I20260811 22:20:37.871829 5617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d370954d216c4511a079e55ec47ec5cd" candidate_uuid: "3e9da459ce8e46ca98cfd45c509c9031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef19082422a428bbb516d98c1e97188" is_pre_election: true
I20260811 22:20:37.872036 5395 raft_consensus.cc:2454] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e9da459ce8e46ca98cfd45c509c9031 in term 0.
I20260811 22:20:37.872134 5617 raft_consensus.cc:2454] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e9da459ce8e46ca98cfd45c509c9031 in term 0.
I20260811 22:20:37.872521 5035 leader_election.cc:302] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e9da459ce8e46ca98cfd45c509c9031, cffe2acce7864676947d85f51fca87b8; no voters:
I20260811 22:20:37.872788 5749 raft_consensus.cc:2781] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:37.872884 5749 raft_consensus.cc:482] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:37.872936 5749 raft_consensus.cc:3037] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:37.873993 5749 raft_consensus.cc:504] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.874389 5749 leader_election.cc:288] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [CANDIDATE]: Term 1 election: Requested vote from peers cffe2acce7864676947d85f51fca87b8 (127.29.3.66:33567), 2ef19082422a428bbb516d98c1e97188 (127.29.3.67:45203)
I20260811 22:20:37.874730 5395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d370954d216c4511a079e55ec47ec5cd" candidate_uuid: "3e9da459ce8e46ca98cfd45c509c9031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffe2acce7864676947d85f51fca87b8"
I20260811 22:20:37.874847 5395 raft_consensus.cc:3037] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:37.874895 5617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d370954d216c4511a079e55ec47ec5cd" candidate_uuid: "3e9da459ce8e46ca98cfd45c509c9031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef19082422a428bbb516d98c1e97188"
I20260811 22:20:37.875026 5617 raft_consensus.cc:3037] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:37.876008 5395 raft_consensus.cc:2454] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e9da459ce8e46ca98cfd45c509c9031 in term 1.
I20260811 22:20:37.876050 5617 raft_consensus.cc:2454] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e9da459ce8e46ca98cfd45c509c9031 in term 1.
I20260811 22:20:37.876286 5035 leader_election.cc:302] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e9da459ce8e46ca98cfd45c509c9031, cffe2acce7864676947d85f51fca87b8; no voters:
I20260811 22:20:37.876446 5749 raft_consensus.cc:2781] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:37.876706 5749 raft_consensus.cc:686] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9da459ce8e46ca98cfd45c509c9031, State: Running, Role: LEADER
I20260811 22:20:37.877053 5749 consensus_queue.cc:237] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } }
I20260811 22:20:37.880239 4871 catalog_manager.cc:5495] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e9da459ce8e46ca98cfd45c509c9031 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "3e9da459ce8e46ca98cfd45c509c9031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e9da459ce8e46ca98cfd45c509c9031" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35023 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 } health_report { overall_health: UNKNOWN } } }
W20260811 22:20:37.893622 5241 tablet.cc:2369] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:37.927654 5739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.924154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.29.3.65:35023)
0811 22:20:37.924289 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:37.924294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:37.924309 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:37.924420 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:37.924423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:37.924481 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:37.924574 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:37.924846 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.925357 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.926203 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:37.926443 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:37.926533 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:37.926853 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:37.927274 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:37.927382 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:37.927458 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 22:20:37.927517 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:20:37.933756 5617 raft_consensus.cc:1261] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 1 FOLLOWER]: Refusing update from remote peer 3e9da459ce8e46ca98cfd45c509c9031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:37.933764 5395 raft_consensus.cc:1261] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 1 FOLLOWER]: Refusing update from remote peer 3e9da459ce8e46ca98cfd45c509c9031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:37.934463 5749 consensus_queue.cc:1035] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef19082422a428bbb516d98c1e97188" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 45203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:37.934686 5760 consensus_queue.cc:1035] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffe2acce7864676947d85f51fca87b8" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:37.952319 4872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:37.952486 4872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:37.954840 4872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0a7d778d4dc4fa3bbcf255bbe28289b: Sending DeleteTablet for 3 replicas of tablet d370954d216c4511a079e55ec47ec5cd
I20260811 22:20:37.955693 5366 tablet_service.cc:1514] Processing DeleteTablet for tablet d370954d216c4511a079e55ec47ec5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:37 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:35216
I20260811 22:20:37.955798 5116 tablet_service.cc:1514] Processing DeleteTablet for tablet d370954d216c4511a079e55ec47ec5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:37 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39466
I20260811 22:20:37.955912 5586 tablet_service.cc:1514] Processing DeleteTablet for tablet d370954d216c4511a079e55ec47ec5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:37 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40350
I20260811 22:20:37.956362 5780 tablet_replica.cc:330] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: stopping tablet replica
I20260811 22:20:37.956374 5777 tablet_replica.cc:330] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: stopping tablet replica
I20260811 22:20:37.956611 5779 tablet_replica.cc:330] T d370954d216c4511a079e55ec47ec5cd P 3e9da459ce8e46ca98cfd45c509c9031: stopping tablet replica
I20260811 22:20:37.956737 5777 raft_consensus.cc:2229] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:37.956866 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4999
I20260811 22:20:37.961802 5780 raft_consensus.cc:2229] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:37.962201 5780 raft_consensus.cc:2258] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:37.962208 5777 raft_consensus.cc:2258] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:37.964929 5780 ts_tablet_manager.cc:1905] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:37.966799 5777 ts_tablet_manager.cc:1905] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:37.967718 5780 ts_tablet_manager.cc:1918] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:37.967902 5780 log.cc:1196] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/d370954d216c4511a079e55ec47ec5cd
I20260811 22:20:37.968322 5780 ts_tablet_manager.cc:1939] T d370954d216c4511a079e55ec47ec5cd P 2ef19082422a428bbb516d98c1e97188: Deleting consensus metadata
I20260811 22:20:37.969595 5777 ts_tablet_manager.cc:1918] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:37.969705 5777 log.cc:1196] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/d370954d216c4511a079e55ec47ec5cd
I20260811 22:20:37.969856 4852 catalog_manager.cc:4841] TS 2ef19082422a428bbb516d98c1e97188 (127.29.3.67:45203): tablet d370954d216c4511a079e55ec47ec5cd (table test-table [id=c9d5bccd94244eeaa97588ff2838d423]) successfully deleted
I20260811 22:20:37.969987 5777 ts_tablet_manager.cc:1939] T d370954d216c4511a079e55ec47ec5cd P cffe2acce7864676947d85f51fca87b8: Deleting consensus metadata
I20260811 22:20:37.971132 4852 catalog_manager.cc:4841] TS cffe2acce7864676947d85f51fca87b8 (127.29.3.66:33567): tablet d370954d216c4511a079e55ec47ec5cd (table test-table [id=c9d5bccd94244eeaa97588ff2838d423]) successfully deleted
W20260811 22:20:37.974046 4852 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:35023 (error 108)
I20260811 22:20:37.974179 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5246
I20260811 22:20:37.979773 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:37.979431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35023 (local address 127.0.0.1:42770)
0811 22:20:37.979613 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:37.979688 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:35023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260811 22:20:37.980057 4852 catalog_manager.cc:4584] TS 3e9da459ce8e46ca98cfd45c509c9031 (127.29.3.65:35023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d370954d216c4511a079e55ec47ec5cd: Network error: Client connection negotiation failed: client connection to 127.29.3.65:35023: connect: Connection refused (error 111)
I20260811 22:20:37.992762 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5492
I20260811 22:20:38.024762 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4818
2026-08-11T22:20:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2660 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T22:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:38Z Disabled control of system clock
I20260811 22:20:38.063153 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42411
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:42411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 22:20:38.175222 5791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:38.175477 5791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:38.175513 5791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:38.179490 5791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:38.179543 5791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:38.179556 5791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:38.179584 5791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:38.179597 5791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:38.184262 5791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:42411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:38.185314 5791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:38.186717 5791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:38.195050 5805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:38.195075 5804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:38.195129 5791 server_base.cc:1034] running on GCE node
W20260811 22:20:38.195025 5808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:38.195852 5791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:38.196913 5791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:38.198082 5791 hybrid_clock.cc:648] HybridClock initialized: now 1786486838198056 us; error 46 us; skew 500 ppm
I20260811 22:20:38.199867 5791 webserver.cc:460] Webserver started at http://127.29.3.126:45857/ using document root <none> and password file <none>
I20260811 22:20:38.200423 5791 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:38.200491 5791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:38.200685 5791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:38.202746 5791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "78a25c7b0cd44b6d96a97e948992c0fb"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "a59b8867f84a91693e15b9434df23797"
server_key_iv: "64d8510b7929026d108bb7c8d09d3b9c"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.203289 5791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "78a25c7b0cd44b6d96a97e948992c0fb"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "a59b8867f84a91693e15b9434df23797"
server_key_iv: "64d8510b7929026d108bb7c8d09d3b9c"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.206842 5791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:38.209146 5819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.210382 5791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:38.210495 5791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "78a25c7b0cd44b6d96a97e948992c0fb"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "a59b8867f84a91693e15b9434df23797"
server_key_iv: "64d8510b7929026d108bb7c8d09d3b9c"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.210588 5791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:38.215219 5791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:38.215811 5791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:38.216001 5791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:38.222518 5791 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:42411
I20260811 22:20:38.222541 5923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:42411 every 8 connection(s)
I20260811 22:20:38.223428 5791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:38.226589 5924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:38.229025 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5791
I20260811 22:20:38.229151 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:38.229461 29709 external_mini_cluster.cc:1428] Setting key 8fb1a24dd260bb43143f936967d81dbd
I20260811 22:20:38.232580 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb: Bootstrap starting.
I20260811 22:20:38.234938 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.230894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.29.3.126:42411)
0811 22:20:38.231509 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.231524 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.231561 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:20:38.232193 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.232210 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.232559 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.232864 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.233675 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.233680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.233692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.233702 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.233756 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.233772 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.233877 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.233978 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.234213 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
I20260811 22:20:38.235045 5924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:38.235785 5924 log.cc:824] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:38.237548 5924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb: No bootstrap required, opened a new log
I20260811 22:20:38.240912 5924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42411 } }
I20260811 22:20:38.241124 5924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:38.241181 5924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a25c7b0cd44b6d96a97e948992c0fb, State: Initialized, Role: FOLLOWER
I20260811 22:20:38.241590 5924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42411 } }
I20260811 22:20:38.241729 5924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:38.241797 5924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:38.241895 5924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:38.242772 5924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42411 } }
I20260811 22:20:38.243080 5924 leader_election.cc:302] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78a25c7b0cd44b6d96a97e948992c0fb; no voters:
I20260811 22:20:38.243347 5924 leader_election.cc:288] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:38.243431 5933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:38.243667 5933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [term 1 LEADER]: Becoming Leader. State: Replica: 78a25c7b0cd44b6d96a97e948992c0fb, State: Running, Role: LEADER
I20260811 22:20:38.243988 5933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42411 } }
I20260811 22:20:38.244488 5924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:38.245507 5935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42411 } } }
I20260811 22:20:38.245635 5935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:38.245764 5936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78a25c7b0cd44b6d96a97e948992c0fb. Latest consensus state: current_term: 1 leader_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78a25c7b0cd44b6d96a97e948992c0fb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42411 } } }
I20260811 22:20:38.246115 5936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:38.246879 5943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:38.249398 5943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:38.254679 5943 catalog_manager.cc:1347] Generated new cluster ID: b10ee9170f444dcd8684b8dec474f90a
I20260811 22:20:38.254741 5943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:38.264623 5943 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:38.264936 5943 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:38.270236 5943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb: Generated new TSK 0
I20260811 22:20:38.270941 5943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:20:38.273996 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42411
--builtin_ntp_servers=127.29.3.84:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 22:20:38.386947 5960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:38.387216 5960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:38.387271 5960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:38.391414 5960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:38.391489 5960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:38.391562 5960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:38.396350 5960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:38.397369 5960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:38.398751 5960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:38.408540 5968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:38.408540 5970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:38.408792 5960 server_base.cc:1034] running on GCE node
W20260811 22:20:38.408938 5966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:38.409402 5960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:38.410096 5960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:38.411263 5960 hybrid_clock.cc:648] HybridClock initialized: now 1786486838411245 us; error 51 us; skew 500 ppm
I20260811 22:20:38.413288 5960 webserver.cc:460] Webserver started at http://127.29.3.65:35861/ using document root <none> and password file <none>
I20260811 22:20:38.413954 5960 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:38.414021 5960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:38.414219 5960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:38.416455 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "dd186d0fad6b4050bc2cd19bfd93db01"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "7531b37dfa6ddc51e32c77c43e74d059"
server_key_iv: "4e1ee107658b3c9ea3418942820c5da3"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.416973 5960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "dd186d0fad6b4050bc2cd19bfd93db01"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "7531b37dfa6ddc51e32c77c43e74d059"
server_key_iv: "4e1ee107658b3c9ea3418942820c5da3"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.420728 5960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:38.423267 5980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.424500 5960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:38.424621 5960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "dd186d0fad6b4050bc2cd19bfd93db01"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "7531b37dfa6ddc51e32c77c43e74d059"
server_key_iv: "4e1ee107658b3c9ea3418942820c5da3"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.424733 5960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:38.429385 5960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:38.429999 5960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:38.430171 5960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:38.430768 5960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:38.431710 5960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:38.431758 5960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.431800 5960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:38.431829 5960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.440155 5960 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:34321
I20260811 22:20:38.440178 6172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:34321 every 8 connection(s)
I20260811 22:20:38.441098 5960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:38.447199 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.442950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:33859 (local address 127.29.3.126:42411)
0811 22:20:38.443096 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.443100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.444204 (+ 1104us) server_negotiation.cc:407] Connection header received
0811 22:20:38.445654 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.445657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.445697 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.445769 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.446820 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.446823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.446825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.446836 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.446857 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.446865 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.446941 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.447047 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.447084 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:20:38.448004 6176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.443332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42411 (local address 127.29.3.65:33859)
0811 22:20:38.444052 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.444094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.445468 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.445957 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.445964 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.446055 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.446564 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.446565 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.446605 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.446673 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.446944 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.446948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.446953 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.447078 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.447271 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260811 22:20:38.449088 6173 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42411
I20260811 22:20:38.449306 6173 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:38.449689 6173 heartbeater.cc:511] Master 127.29.3.126:42411 requested a full tablet report, sending...
I20260811 22:20:38.451107 5851 ts_manager.cc:194] Registered new tserver with Master: dd186d0fad6b4050bc2cd19bfd93db01 (127.29.3.65:34321)
I20260811 22:20:38.451143 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5960
I20260811 22:20:38.451246 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:38.451517 29709 external_mini_cluster.cc:1428] Setting key 5f1b9957d047f67bc9065dee145efa73
I20260811 22:20:38.453253 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42411
--builtin_ntp_servers=127.29.3.84:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 22:20:38.565688 6177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:38.565953 6177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:38.565990 6177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:38.570016 6177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:38.570080 6177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:38.570148 6177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:38.575062 6177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:38.576192 6177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:38.577618 6177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:38.587283 6186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:38.587283 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:38.587327 6188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:38.587358 6177 server_base.cc:1034] running on GCE node
I20260811 22:20:38.587924 6177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:38.588565 6177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:38.589730 6177 hybrid_clock.cc:648] HybridClock initialized: now 1786486838589712 us; error 27 us; skew 500 ppm
I20260811 22:20:38.591498 6177 webserver.cc:460] Webserver started at http://127.29.3.66:43089/ using document root <none> and password file <none>
I20260811 22:20:38.592060 6177 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:38.592121 6177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:38.592330 6177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:38.594409 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "8bb91b45ab2d40a2be75292776d8440a"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "1357598a0ed3951071847094a2c43a34"
server_key_iv: "e0e4bef4b6b9770d4765f43392804ee6"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.594956 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "8bb91b45ab2d40a2be75292776d8440a"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "1357598a0ed3951071847094a2c43a34"
server_key_iv: "e0e4bef4b6b9770d4765f43392804ee6"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.598558 6177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:38.601030 6199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.602239 6177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:38.602377 6177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "8bb91b45ab2d40a2be75292776d8440a"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "1357598a0ed3951071847094a2c43a34"
server_key_iv: "e0e4bef4b6b9770d4765f43392804ee6"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.602487 6177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:38.607028 6177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:38.607611 6177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:38.607776 6177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:38.608299 6177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:38.609191 6177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:38.609241 6177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.609282 6177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:38.609310 6177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.617693 6177 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:36491
I20260811 22:20:38.617729 6401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:36491 every 8 connection(s)
I20260811 22:20:38.618611 6177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:38.620179 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 6177
I20260811 22:20:38.620293 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:38.620599 29709 external_mini_cluster.cc:1428] Setting key 397d73a024f9bf3a5bae5abe88ee101e
I20260811 22:20:38.622700 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42411
--builtin_ntp_servers=127.29.3.84:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 22:20:38.624812 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.620393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:37293 (local address 127.29.3.126:42411)
0811 22:20:38.620565 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.620571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.621712 (+ 1141us) server_negotiation.cc:407] Connection header received
0811 22:20:38.623236 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.623240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.623284 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.623367 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.624406 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.624410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.624412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.624426 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.624449 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.624457 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.624528 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.624672 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.624708 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 22:20:38.625965 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.620828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42411 (local address 127.29.3.66:37293)
0811 22:20:38.621553 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.621607 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.623019 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.623492 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.623498 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.623586 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.624112 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.624114 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.624154 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.624225 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.624534 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.624542 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.624547 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.624732 (+ 185us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.624984 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":103,"threads_started":1}
I20260811 22:20:38.627053 6402 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42411
I20260811 22:20:38.627296 6402 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:38.627720 6402 heartbeater.cc:511] Master 127.29.3.126:42411 requested a full tablet report, sending...
I20260811 22:20:38.628734 5851 ts_manager.cc:194] Registered new tserver with Master: 8bb91b45ab2d40a2be75292776d8440a (127.29.3.66:36491)
W20260811 22:20:38.738183 6406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:38.738545 6406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:38.738608 6406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:38.742738 6406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:38.742800 6406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:38.742877 6406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:38.747946 6406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:38.749027 6406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:38.750454 6406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:38.760181 6413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:38.760180 6415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:38.760236 6411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:38.760402 6406 server_base.cc:1034] running on GCE node
I20260811 22:20:38.760882 6406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:38.761586 6406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:38.762755 6406 hybrid_clock.cc:648] HybridClock initialized: now 1786486838762736 us; error 30 us; skew 500 ppm
I20260811 22:20:38.764678 6406 webserver.cc:460] Webserver started at http://127.29.3.67:46009/ using document root <none> and password file <none>
I20260811 22:20:38.765293 6406 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:38.765369 6406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:38.765570 6406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:38.767761 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "64a6720526744766be748adf2e1dfc92"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "5bdf50fc8dbd129b404b6c7615e07574"
server_key_iv: "a380326db7ee8ae854ebf4ab0ba58e98"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.768340 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "64a6720526744766be748adf2e1dfc92"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "5bdf50fc8dbd129b404b6c7615e07574"
server_key_iv: "a380326db7ee8ae854ebf4ab0ba58e98"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.772224 6406 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:38.774909 6428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.776188 6406 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:38.776321 6406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "64a6720526744766be748adf2e1dfc92"
format_stamp: "Formatted at 2026-08-11 22:20:38 on dist-test-slave-4tdj"
server_key: "5bdf50fc8dbd129b404b6c7615e07574"
server_key_iv: "a380326db7ee8ae854ebf4ab0ba58e98"
server_key_version: "encryptionkey@0"
I20260811 22:20:38.776433 6406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:38.781095 6406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:38.781780 6406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:38.781946 6406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:38.782582 6406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:38.783564 6406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:38.783613 6406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.783665 6406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:38.783695 6406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:38.792142 6406 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:41675
I20260811 22:20:38.792199 6612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:41675 every 8 connection(s)
I20260811 22:20:38.793059 6406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:38.799046 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.794848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:54193 (local address 127.29.3.126:42411)
0811 22:20:38.794974 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.794977 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.796031 (+ 1054us) server_negotiation.cc:407] Connection header received
0811 22:20:38.797339 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.797344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.797393 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.797470 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.798580 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.798584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.798587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.798602 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.798623 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.798631 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.798719 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.798851 (+ 132us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.798898 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:20:38.799849 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 6406
I20260811 22:20:38.799877 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.795230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42411 (local address 127.29.3.67:54193)
0811 22:20:38.795880 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.795922 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.797098 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.797647 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.797653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.797747 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.798267 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.798268 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.798357 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.798429 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.798709 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.798719 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.798725 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.798885 (+ 160us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.799126 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":100,"threads_started":1}
I20260811 22:20:38.799952 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:38.800185 29709 external_mini_cluster.cc:1428] Setting key 71f57ad6a79738b16a61465c3fca5f5e
I20260811 22:20:38.801039 6613 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42411
I20260811 22:20:38.801276 6613 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:38.801723 6613 heartbeater.cc:511] Master 127.29.3.126:42411 requested a full tablet report, sending...
I20260811 22:20:38.802770 5851 ts_manager.cc:194] Registered new tserver with Master: 64a6720526744766be748adf2e1dfc92 (127.29.3.67:41675)
I20260811 22:20:38.813699 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:20:38.816985 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.815633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.29.3.126:42411)
0811 22:20:38.815771 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.815776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.815900 (+ 124us) server_negotiation.cc:407] Connection header received
0811 22:20:38.815998 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.816000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.816033 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.816086 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.816679 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.816681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.816683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.816694 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.816712 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.816720 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.816784 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.816831 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.816862 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:38.819768 5851 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:38.822760 5851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:38.833715 6625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.830932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:41675 (local address 127.0.0.1:40142)
0811 22:20:38.831424 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.831471 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.831566 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.832280 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.832287 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.832367 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.832906 (+ 539us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.832908 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.832930 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.832999 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.833376 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.833387 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.833392 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.833507 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.833574 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":84,"threads_started":1}
I20260811 22:20:38.834376 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.831058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.29.3.67:41675)
0811 22:20:38.831809 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.831820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.831847 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:20:38.831903 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.831909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.832104 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.832265 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.833138 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.833144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.833157 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.833169 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.833215 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.833253 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.833401 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.834085 (+ 684us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.834189 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":177,"threads_started":1}
I20260811 22:20:38.834780 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.831088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34321 (local address 127.0.0.1:41442)
0811 22:20:38.831751 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.831764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.831833 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.833160 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.833163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.833178 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.833590 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.833592 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.833604 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.833636 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.834467 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.834471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.834474 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.834527 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.834559 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":113,"threads_started":1}
I20260811 22:20:38.834784 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.831655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:36491 (local address 127.0.0.1:48888)
0811 22:20:38.832166 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.832176 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.832251 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.833222 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.833225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.833236 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.833739 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.833741 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.833751 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.833779 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.834481 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.834484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.834488 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.834543 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.834578 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":90,"threads_started":1}
I20260811 22:20:38.834920 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.832037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.29.3.65:34321)
0811 22:20:38.832499 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.832510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.832535 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:20:38.832874 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.832886 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.833043 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.833194 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.834234 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.834240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.834251 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.834262 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.834334 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.834369 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.834663 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.834713 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.834779 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":140,"threads_started":1}
I20260811 22:20:38.835223 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.832429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.29.3.66:36491)
0811 22:20:38.832813 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.832824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.832847 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:20:38.832901 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.832906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.833091 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.833419 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.834252 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.834258 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.834269 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.834279 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.834339 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.834364 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.834964 (+ 600us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.835014 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.835083 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":94,"threads_started":1}
I20260811 22:20:38.837679 6508 tablet_service.cc:1467] Processing CreateTablet for tablet 02ef751124fc4a4297b8cc8b01c56f70 (DEFAULT_TABLE table=test-table [id=5c7ec4d7d6984ec9bc561ca74a9e6368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:38.837821 6276 tablet_service.cc:1467] Processing CreateTablet for tablet 02ef751124fc4a4297b8cc8b01c56f70 (DEFAULT_TABLE table=test-table [id=5c7ec4d7d6984ec9bc561ca74a9e6368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:38.837821 6063 tablet_service.cc:1467] Processing CreateTablet for tablet 02ef751124fc4a4297b8cc8b01c56f70 (DEFAULT_TABLE table=test-table [id=5c7ec4d7d6984ec9bc561ca74a9e6368]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:38.838824 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ef751124fc4a4297b8cc8b01c56f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:38.838894 6276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ef751124fc4a4297b8cc8b01c56f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:38.838894 6063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02ef751124fc4a4297b8cc8b01c56f70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:38.845270 6636 tablet_bootstrap.cc:492] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Bootstrap starting.
I20260811 22:20:38.846690 6639 tablet_bootstrap.cc:492] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Bootstrap starting.
I20260811 22:20:38.847669 6638 tablet_bootstrap.cc:492] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: Bootstrap starting.
I20260811 22:20:38.847782 6636 tablet_bootstrap.cc:654] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:38.848594 6639 tablet_bootstrap.cc:654] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:38.848728 6636 log.cc:824] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:38.849372 6639 log.cc:824] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:38.849748 6638 tablet_bootstrap.cc:654] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:38.850570 6638 log.cc:824] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:38.851002 6636 tablet_bootstrap.cc:492] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: No bootstrap required, opened a new log
I20260811 22:20:38.851146 6639 tablet_bootstrap.cc:492] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: No bootstrap required, opened a new log
I20260811 22:20:38.851192 6636 ts_tablet_manager.cc:1397] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260811 22:20:38.851344 6639 ts_tablet_manager.cc:1397] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 22:20:38.852530 6638 tablet_bootstrap.cc:492] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: No bootstrap required, opened a new log
I20260811 22:20:38.852705 6638 ts_tablet_manager.cc:1397] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 22:20:38.855033 6639 raft_consensus.cc:348] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.855263 6639 raft_consensus.cc:374] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:38.855337 6639 raft_consensus.cc:729] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb91b45ab2d40a2be75292776d8440a, State: Initialized, Role: FOLLOWER
I20260811 22:20:38.855585 6636 raft_consensus.cc:348] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.855778 6639 consensus_queue.cc:260] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.855870 6636 raft_consensus.cc:374] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:38.855967 6636 raft_consensus.cc:729] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a6720526744766be748adf2e1dfc92, State: Initialized, Role: FOLLOWER
I20260811 22:20:38.856408 6636 consensus_queue.cc:260] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.856694 6639 ts_tablet_manager.cc:1428] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 22:20:38.857102 6402 heartbeater.cc:503] Master 127.29.3.126:42411 was elected leader, sending a full tablet report...
I20260811 22:20:38.857105 6613 heartbeater.cc:503] Master 127.29.3.126:42411 was elected leader, sending a full tablet report...
I20260811 22:20:38.856992 6638 raft_consensus.cc:348] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.857290 6638 raft_consensus.cc:374] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:38.857360 6638 raft_consensus.cc:729] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd186d0fad6b4050bc2cd19bfd93db01, State: Initialized, Role: FOLLOWER
I20260811 22:20:38.857761 6638 consensus_queue.cc:260] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.858712 6638 ts_tablet_manager.cc:1428] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:20:38.858721 6636 ts_tablet_manager.cc:1428] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260811 22:20:38.859076 6173 heartbeater.cc:503] Master 127.29.3.126:42411 was elected leader, sending a full tablet report...
W20260811 22:20:38.869590 6403 tablet.cc:2369] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:38.887039 6647 raft_consensus.cc:482] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:38.887293 6647 raft_consensus.cc:504] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.888497 6647 leader_election.cc:288] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd186d0fad6b4050bc2cd19bfd93db01 (127.29.3.65:34321), 8bb91b45ab2d40a2be75292776d8440a (127.29.3.66:36491)
I20260811 22:20:38.890021 6645 raft_consensus.cc:482] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:38.890241 6645 raft_consensus.cc:504] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.890528 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.888803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34321 (local address 127.29.3.67:43039)
0811 22:20:38.888945 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.888963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.889063 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.889342 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.889346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.889364 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.889890 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.889893 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.889906 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.889938 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.890262 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.890265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.890268 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.890367 (+ 99us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.890404 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 22:20:38.890839 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.888918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43039 (local address 127.29.3.65:34321)
0811 22:20:38.889052 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.889056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.889072 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:20:38.889170 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.889173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.889214 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.889290 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.890087 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.890091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.890093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.890105 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.890126 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.890137 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.890222 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.890676 (+ 454us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.890724 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:20:38.891088 6635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.889256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:59281 (local address 127.29.3.66:36491)
0811 22:20:38.889391 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.889396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.889646 (+ 250us) server_negotiation.cc:407] Connection header received
0811 22:20:38.889742 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.889766 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.889810 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.889896 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.890646 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.890650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.890653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.890677 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.890698 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.890708 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.890768 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.890928 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.890970 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:20:38.891455 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.889110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:36491 (local address 127.29.3.67:59281)
0811 22:20:38.889541 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.889555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.889649 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.889950 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.889953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.889989 (+ 36us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.890473 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.890475 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.890485 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.890513 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.890835 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.890837 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.890840 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.891286 (+ 446us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.891322 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":154,"threads_started":1}
I20260811 22:20:38.891585 6092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ef751124fc4a4297b8cc8b01c56f70" candidate_uuid: "64a6720526744766be748adf2e1dfc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" is_pre_election: true
I20260811 22:20:38.891878 6092 raft_consensus.cc:2454] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64a6720526744766be748adf2e1dfc92 in term 0.
I20260811 22:20:38.892393 6430 leader_election.cc:302] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a6720526744766be748adf2e1dfc92, dd186d0fad6b4050bc2cd19bfd93db01; no voters:
I20260811 22:20:38.892683 6647 raft_consensus.cc:2781] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:38.892791 6647 raft_consensus.cc:482] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:38.892843 6647 raft_consensus.cc:3037] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:38.893004 6645 leader_election.cc:288] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64a6720526744766be748adf2e1dfc92 (127.29.3.67:41675), dd186d0fad6b4050bc2cd19bfd93db01 (127.29.3.65:34321)
I20260811 22:20:38.893193 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ef751124fc4a4297b8cc8b01c56f70" candidate_uuid: "64a6720526744766be748adf2e1dfc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb91b45ab2d40a2be75292776d8440a" is_pre_election: true
I20260811 22:20:38.893388 6309 raft_consensus.cc:2454] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64a6720526744766be748adf2e1dfc92 in term 0.
I20260811 22:20:38.894695 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.892280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:53153 (local address 127.29.3.67:41675)
0811 22:20:38.892801 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.892805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.893329 (+ 524us) server_negotiation.cc:407] Connection header received
0811 22:20:38.893483 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.893486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.893519 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.893566 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.894343 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.894346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.894348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.894361 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.894379 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.894389 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.894472 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.894556 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.894592 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454}
I20260811 22:20:38.894748 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.891789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:41675 (local address 127.29.3.66:53153)
0811 22:20:38.893243 (+ 1454us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.893256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.893382 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.893684 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.893686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.893701 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.894140 (+ 439us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.894142 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.894153 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.894178 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.894462 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.894466 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.894469 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.894555 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.894589 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1377,"mutex_wait_us":847,"thread_start_us":137,"threads_started":1}
I20260811 22:20:38.894763 6647 raft_consensus.cc:504] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.895042 6404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.891790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34321 (local address 127.29.3.66:39169)
0811 22:20:38.892065 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.892079 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:38.892143 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:38.894139 (+ 1996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:38.894142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:38.894157 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:38.894529 (+ 372us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:20:38.894531 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:38.894541 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:20:38.894564 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:38.894836 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:38.894839 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:38.894842 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 22:20:38.894911 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 22:20:38.894944 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260811 22:20:38.895056 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.892580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:39169 (local address 127.29.3.65:34321)
0811 22:20:38.893929 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.893933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.893946 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:38.893980 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.893983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.894017 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.894094 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.894678 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.894681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.894684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.894696 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.894715 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.894725 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.894798 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.894916 (+ 118us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.894949 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266}
I20260811 22:20:38.895104 6647 leader_election.cc:288] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [CANDIDATE]: Term 1 election: Requested vote from peers dd186d0fad6b4050bc2cd19bfd93db01 (127.29.3.65:34321), 8bb91b45ab2d40a2be75292776d8440a (127.29.3.66:36491)
I20260811 22:20:38.895270 6538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ef751124fc4a4297b8cc8b01c56f70" candidate_uuid: "8bb91b45ab2d40a2be75292776d8440a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64a6720526744766be748adf2e1dfc92" is_pre_election: true
I20260811 22:20:38.895495 6538 raft_consensus.cc:2379] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8bb91b45ab2d40a2be75292776d8440a in current term 1: Already voted for candidate 64a6720526744766be748adf2e1dfc92 in this term.
I20260811 22:20:38.895526 6092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ef751124fc4a4297b8cc8b01c56f70" candidate_uuid: "64a6720526744766be748adf2e1dfc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd186d0fad6b4050bc2cd19bfd93db01"
I20260811 22:20:38.895565 6309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ef751124fc4a4297b8cc8b01c56f70" candidate_uuid: "64a6720526744766be748adf2e1dfc92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bb91b45ab2d40a2be75292776d8440a"
I20260811 22:20:38.895659 6309 raft_consensus.cc:3037] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:38.895638 6092 raft_consensus.cc:3037] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:38.896136 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02ef751124fc4a4297b8cc8b01c56f70" candidate_uuid: "8bb91b45ab2d40a2be75292776d8440a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" is_pre_election: true
I20260811 22:20:38.896765 6201 leader_election.cc:302] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8bb91b45ab2d40a2be75292776d8440a; no voters: 64a6720526744766be748adf2e1dfc92, dd186d0fad6b4050bc2cd19bfd93db01
I20260811 22:20:38.896849 6309 raft_consensus.cc:2454] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64a6720526744766be748adf2e1dfc92 in term 1.
I20260811 22:20:38.896943 6092 raft_consensus.cc:2454] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64a6720526744766be748adf2e1dfc92 in term 1.
I20260811 22:20:38.897043 6645 raft_consensus.cc:2726] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 22:20:38.897128 6430 leader_election.cc:302] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64a6720526744766be748adf2e1dfc92, 8bb91b45ab2d40a2be75292776d8440a; no voters:
I20260811 22:20:38.897318 6647 raft_consensus.cc:2781] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:38.897578 6647 raft_consensus.cc:686] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 1 LEADER]: Becoming Leader. State: Replica: 64a6720526744766be748adf2e1dfc92, State: Running, Role: LEADER
I20260811 22:20:38.897903 6647 consensus_queue.cc:237] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } }
I20260811 22:20:38.901284 5849 catalog_manager.cc:5495] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64a6720526744766be748adf2e1dfc92 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "64a6720526744766be748adf2e1dfc92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64a6720526744766be748adf2e1dfc92" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 41675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:38.915452 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.913818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.29.3.67:41675)
0811 22:20:38.913926 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:38.913930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:38.914042 (+ 112us) server_negotiation.cc:407] Connection header received
0811 22:20:38.914229 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:38.914232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:38.914319 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:38.914403 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:38.915054 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:38.915059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:38.915062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:38.915076 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:38.915099 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:38.915109 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:38.915202 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:38.915308 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 22:20:38.915346 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 22:20:38.921509 6092 raft_consensus.cc:1261] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 1 FOLLOWER]: Refusing update from remote peer 64a6720526744766be748adf2e1dfc92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:38.921579 6309 raft_consensus.cc:1261] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 1 FOLLOWER]: Refusing update from remote peer 64a6720526744766be748adf2e1dfc92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:38.922199 6655 consensus_queue.cc:1035] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd186d0fad6b4050bc2cd19bfd93db01" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:38.922417 6647 consensus_queue.cc:1035] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bb91b45ab2d40a2be75292776d8440a" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:38.940447 5849 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:38.940657 5849 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:38.943462 5849 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 78a25c7b0cd44b6d96a97e948992c0fb: Sending DeleteTablet for 3 replicas of tablet 02ef751124fc4a4297b8cc8b01c56f70
I20260811 22:20:38.944252 6508 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ef751124fc4a4297b8cc8b01c56f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:38 UTC) from {username='slave'} at 127.0.0.1:40142
I20260811 22:20:38.944442 6063 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ef751124fc4a4297b8cc8b01c56f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:38 UTC) from {username='slave'} at 127.0.0.1:41442
I20260811 22:20:38.945034 6276 tablet_service.cc:1514] Processing DeleteTablet for tablet 02ef751124fc4a4297b8cc8b01c56f70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:38 UTC) from {username='slave'} at 127.0.0.1:48888
I20260811 22:20:38.945171 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5960
I20260811 22:20:38.945206 6671 tablet_replica.cc:330] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01: stopping tablet replica
I20260811 22:20:38.945528 6671 raft_consensus.cc:2229] T 02ef751124fc4a4297b8cc8b01c56f70 P dd186d0fad6b4050bc2cd19bfd93db01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:38.949203 6673 tablet_replica.cc:330] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: stopping tablet replica
I20260811 22:20:38.949604 6673 raft_consensus.cc:2229] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:38.950022 6673 raft_consensus.cc:2258] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:38.953275 6670 tablet_replica.cc:330] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: stopping tablet replica
I20260811 22:20:38.953673 6670 raft_consensus.cc:2229] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:38.954193 6670 raft_consensus.cc:2258] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:38.956236 6673 ts_tablet_manager.cc:1905] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:38.959314 6673 ts_tablet_manager.cc:1918] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:38.959439 6673 log.cc:1196] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/02ef751124fc4a4297b8cc8b01c56f70
I20260811 22:20:38.959785 6673 ts_tablet_manager.cc:1939] T 02ef751124fc4a4297b8cc8b01c56f70 P 8bb91b45ab2d40a2be75292776d8440a: Deleting consensus metadata
I20260811 22:20:38.961131 6670 ts_tablet_manager.cc:1905] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:38.961369 5821 catalog_manager.cc:4841] TS 8bb91b45ab2d40a2be75292776d8440a (127.29.3.66:36491): tablet 02ef751124fc4a4297b8cc8b01c56f70 (table test-table [id=5c7ec4d7d6984ec9bc561ca74a9e6368]) successfully deleted
W20260811 22:20:38.963901 5821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:34321 (error 108)
I20260811 22:20:38.964103 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 6177
I20260811 22:20:38.977342 6630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:38.976920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34321 (local address 127.0.0.1:41454)
0811 22:20:38.977140 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:38.977233 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260811 22:20:38.977608 6670 ts_tablet_manager.cc:1918] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260811 22:20:38.977895 5821 catalog_manager.cc:4584] TS dd186d0fad6b4050bc2cd19bfd93db01 (127.29.3.65:34321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02ef751124fc4a4297b8cc8b01c56f70: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34321: connect: Connection refused (error 111)
I20260811 22:20:38.978116 6670 log.cc:1196] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/02ef751124fc4a4297b8cc8b01c56f70
I20260811 22:20:38.978680 6670 ts_tablet_manager.cc:1939] T 02ef751124fc4a4297b8cc8b01c56f70 P 64a6720526744766be748adf2e1dfc92: Deleting consensus metadata
I20260811 22:20:38.980243 5820 catalog_manager.cc:4841] TS 64a6720526744766be748adf2e1dfc92 (127.29.3.67:41675): tablet 02ef751124fc4a4297b8cc8b01c56f70 (table test-table [id=5c7ec4d7d6984ec9bc561ca74a9e6368]) successfully deleted
I20260811 22:20:38.983769 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 6406
I20260811 22:20:39.007014 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5791
2026-08-11T22:20:39Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (977 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 22:20:39.032848 29709 mini_oidc.cc:152] Starting JWKS server
I20260811 22:20:39.034543 29709 webserver.cc:460] Webserver started at https://127.0.0.1:40705/ using document root <none> and password file <none>
I20260811 22:20:39.034718 29709 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 22:20:39.034986 29709 webserver.cc:460] Webserver started at http://127.0.0.1:39397/ using document root <none> and password file <none>
2026-08-11T22:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:39Z Disabled control of system clock
I20260811 22:20:39.044872 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:33331
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:42585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:33331
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/testchainca.cert with env {}
W20260811 22:20:39.158545 6683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:39.158830 6683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:39.158870 6683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:39.162966 6683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:39.163028 6683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:39.163043 6683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:39.163058 6683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:39.163072 6683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:20:39.163086 6683 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:20:39.163100 6683 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40705/jwks.json
I20260811 22:20:39.168463 6683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:33331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:33331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:39.169482 6683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:39.170557 6683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:39.178787 6700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:39.178783 6698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:39.178905 6683 server_base.cc:1034] running on GCE node
W20260811 22:20:39.178783 6697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:39.179687 6683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:39.180794 6683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:39.181952 6683 hybrid_clock.cc:648] HybridClock initialized: now 1786486839181940 us; error 25 us; skew 500 ppm
I20260811 22:20:39.183777 6683 webserver.cc:460] Webserver started at http://127.29.3.126:45761/ using document root <none> and password file <none>
I20260811 22:20:39.184343 6683 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:39.184405 6683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:39.184607 6683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:39.186712 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "92b6b20eecf84c46bce2ecc8cd020892"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "8c87fa038f7ddedda2f164a64a5367e9"
server_key_iv: "6676ab6eb4a8e81be315965870fcb713"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.187289 6683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "92b6b20eecf84c46bce2ecc8cd020892"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "8c87fa038f7ddedda2f164a64a5367e9"
server_key_iv: "6676ab6eb4a8e81be315965870fcb713"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.191010 6683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:39.193415 6712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:39.194697 6683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:39.194813 6683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "92b6b20eecf84c46bce2ecc8cd020892"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "8c87fa038f7ddedda2f164a64a5367e9"
server_key_iv: "6676ab6eb4a8e81be315965870fcb713"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.194898 6683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:39.211326 6683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:39.212023 6683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:39.212196 6683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:39.219340 6683 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:33331
I20260811 22:20:39.219355 6800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:33331 every 8 connection(s)
I20260811 22:20:39.220358 6683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:39.221027 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 6683
I20260811 22:20:39.221143 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:39.221418 29709 external_mini_cluster.cc:1428] Setting key a6add029a557f4f788db4e8c60794dc3
I20260811 22:20:39.224076 6801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:39.231029 6801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892: Bootstrap starting.
I20260811 22:20:39.231889 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:39.222820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.29.3.126:33331)
0811 22:20:39.223529 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:39.223551 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:39.223594 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:20:39.224459 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:39.224484 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:39.224995 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:39.225371 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:39.225826 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.227310 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.228170 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.228852 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.229021 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.229789 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:39.229795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:39.229812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:39.230343 (+ 531us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:39.230419 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:39.230445 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:39.230543 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:39.230673 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 22:20:39.230948 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":117,"threads_started":1}
I20260811 22:20:39.233531 6801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:39.234390 6801 log.cc:824] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:39.234745 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:20:39.236328 6801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892: No bootstrap required, opened a new log
I20260811 22:20:39.239800 6801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b6b20eecf84c46bce2ecc8cd020892" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33331 } }
I20260811 22:20:39.240028 6801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:39.240087 6801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b6b20eecf84c46bce2ecc8cd020892, State: Initialized, Role: FOLLOWER
I20260811 22:20:39.240520 6801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b6b20eecf84c46bce2ecc8cd020892" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33331 } }
I20260811 22:20:39.240653 6801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:39.240720 6801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:39.240820 6801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:39.241753 6801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b6b20eecf84c46bce2ecc8cd020892" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33331 } }
I20260811 22:20:39.242080 6801 leader_election.cc:302] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b6b20eecf84c46bce2ecc8cd020892; no voters:
I20260811 22:20:39.242448 6801 leader_election.cc:288] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:39.242540 6812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:39.242787 6812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [term 1 LEADER]: Becoming Leader. State: Replica: 92b6b20eecf84c46bce2ecc8cd020892, State: Running, Role: LEADER
I20260811 22:20:39.243137 6812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b6b20eecf84c46bce2ecc8cd020892" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33331 } }
I20260811 22:20:39.243654 6801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:39.244786 6816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b6b20eecf84c46bce2ecc8cd020892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b6b20eecf84c46bce2ecc8cd020892" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33331 } } }
I20260811 22:20:39.244925 6816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:39.245313 6831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:39.245995 6817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b6b20eecf84c46bce2ecc8cd020892. Latest consensus state: current_term: 1 leader_uuid: "92b6b20eecf84c46bce2ecc8cd020892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b6b20eecf84c46bce2ecc8cd020892" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33331 } } }
I20260811 22:20:39.246080 6817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:39.248129 6831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:39.253437 6831 catalog_manager.cc:1347] Generated new cluster ID: b271a9180b2243729e8649ec39c94612
I20260811 22:20:39.253513 6831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:39.261809 6831 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:39.262789 6831 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:39.270140 6831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92b6b20eecf84c46bce2ecc8cd020892: Generated new TSK 0
I20260811 22:20:39.270864 6831 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 22:20:39.303732 6867 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 22:20:39.303838 6867 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 22:20:39.304636 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:39.286892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.29.3.126:33331)
0811 22:20:39.287031 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:39.287037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:39.287234 (+ 197us) server_negotiation.cc:407] Connection header received
0811 22:20:39.287437 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:39.287440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:39.287502 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:39.287611 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:20:39.287849 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.288686 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.289507 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.289842 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.289939 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.290246 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:20:39.304189 (+ 13943us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 22:20:39.304489 (+ 300us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45188: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 22:20:39.304706 6806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45188: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 22:20:39.305932 6865 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:39.286782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:33331 (local address 127.0.0.1:45188)
0811 22:20:39.287149 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:39.287162 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:39.287281 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:39.287624 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:39.287627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:39.287633 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:20:39.287752 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:39.287757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.288780 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.288783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:39.289405 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:39.289411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.289937 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.289939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:39.290001 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.290048 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:20:39.304429 (+ 14381us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 22:20:39.304636 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.3.126:33331: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":132,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.3.126:33331: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 22:20:39.306838 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 6683
2026-08-11T22:20:39Z chronyd exiting
I20260811 22:20:39.347539 29709 test_util.cc:175] -----------------------------------------------
I20260811 22:20:39.347632 29709 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786486810933177-29709-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (316 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 22:20:39.348750 29709 mini_oidc.cc:152] Starting JWKS server
I20260811 22:20:39.349589 29709 webserver.cc:460] Webserver started at https://127.0.0.1:38317/ using document root <none> and password file <none>
I20260811 22:20:39.349677 29709 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 22:20:39.349962 29709 webserver.cc:460] Webserver started at http://127.0.0.1:39409/ using document root <none> and password file <none>
2026-08-11T22:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:39Z Disabled control of system clock
I20260811 22:20:39.359397 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38547
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:38745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:38547
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/testchainca.cert with env {}
W20260811 22:20:39.469112 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:39.469364 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:39.469419 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:39.473320 6879 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:39.473372 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:39.473394 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:39.473414 6879 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:39.473435 6879 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:20:39.473465 6879 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:20:39.473485 6879 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38317/jwks.json
I20260811 22:20:39.478644 6879 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:38547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:39.479638 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:39.480583 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:39.488575 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:39.488744 6879 server_base.cc:1034] running on GCE node
W20260811 22:20:39.488593 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:39.488704 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:39.489506 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:39.490561 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:39.491709 6879 hybrid_clock.cc:648] HybridClock initialized: now 1786486839491688 us; error 36 us; skew 500 ppm
I20260811 22:20:39.493424 6879 webserver.cc:460] Webserver started at http://127.29.3.126:33691/ using document root <none> and password file <none>
I20260811 22:20:39.493952 6879 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:39.494019 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:39.494210 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:39.496202 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "2b7d1010948041958c4a814b15b89346"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "0bb72be04043edc41fa01c32e37eb803"
server_key_iv: "74929779db008283f34b97ea8fbe236e"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.496811 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "2b7d1010948041958c4a814b15b89346"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "0bb72be04043edc41fa01c32e37eb803"
server_key_iv: "74929779db008283f34b97ea8fbe236e"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.500418 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:20:39.502843 6900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:39.504060 6879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:39.504184 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "2b7d1010948041958c4a814b15b89346"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "0bb72be04043edc41fa01c32e37eb803"
server_key_iv: "74929779db008283f34b97ea8fbe236e"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.504297 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:39.521862 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:39.522665 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:39.522850 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:39.529776 6879 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:38547
I20260811 22:20:39.529798 6986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:38547 every 8 connection(s)
I20260811 22:20:39.530767 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:39.533957 6993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:39.534550 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 6879
I20260811 22:20:39.534694 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:39.535043 29709 external_mini_cluster.cc:1428] Setting key 219d01ca6a69c7ee358a3618c9549229
I20260811 22:20:39.540341 6993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346: Bootstrap starting.
I20260811 22:20:39.542630 6993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:39.543438 6993 log.cc:824] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:39.544075 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:39.536419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.29.3.126:38547)
0811 22:20:39.537093 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:39.537114 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:39.537149 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:20:39.537832 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:39.537849 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:39.538247 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:39.538608 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:39.539027 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.539984 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.540898 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.541362 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.541517 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.542333 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:39.542340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:39.542358 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:39.542739 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:39.542819 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:39.542843 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:39.542931 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:39.543098 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 22:20:39.543355 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":151,"threads_started":1}
I20260811 22:20:39.545540 6993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346: No bootstrap required, opened a new log
I20260811 22:20:39.546892 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:20:39.549481 6993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d1010948041958c4a814b15b89346" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38547 } }
I20260811 22:20:39.549700 6993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:39.549769 6993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b7d1010948041958c4a814b15b89346, State: Initialized, Role: FOLLOWER
I20260811 22:20:39.550172 6993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d1010948041958c4a814b15b89346" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38547 } }
I20260811 22:20:39.550359 6993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:39.550433 6993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:39.550510 6993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:39.551362 6993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d1010948041958c4a814b15b89346" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38547 } }
I20260811 22:20:39.551656 6993 leader_election.cc:302] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b7d1010948041958c4a814b15b89346; no voters:
I20260811 22:20:39.551904 6993 leader_election.cc:288] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:39.552029 7001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:39.552244 7001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [term 1 LEADER]: Becoming Leader. State: Replica: 2b7d1010948041958c4a814b15b89346, State: Running, Role: LEADER
I20260811 22:20:39.552568 7001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d1010948041958c4a814b15b89346" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38547 } }
I20260811 22:20:39.553057 6993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:39.554064 7002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b7d1010948041958c4a814b15b89346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d1010948041958c4a814b15b89346" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38547 } } }
I20260811 22:20:39.554198 7002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:39.554574 7013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:39.554483 7006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b7d1010948041958c4a814b15b89346. Latest consensus state: current_term: 1 leader_uuid: "2b7d1010948041958c4a814b15b89346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b7d1010948041958c4a814b15b89346" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38547 } } }
I20260811 22:20:39.554935 7006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:39.557376 7013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:39.563133 7013 catalog_manager.cc:1347] Generated new cluster ID: 39ead7517e374feca20723dac5caedd8
I20260811 22:20:39.563203 7013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:39.588765 7013 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:39.590101 7013 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:39.610791 7013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b7d1010948041958c4a814b15b89346: Generated new TSK 0
I20260811 22:20:39.611573 7013 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 22:20:39.645169 7049 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 22:20:39.645287 7049 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 22:20:39.645980 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:39.629415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53066 (local address 127.29.3.126:38547)
0811 22:20:39.629559 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:39.629564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:39.629713 (+ 149us) server_negotiation.cc:407] Connection header received
0811 22:20:39.629877 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:39.629879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:39.629927 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:39.630016 (+ 89us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:20:39.630246 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.630799 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.632025 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.632234 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.632302 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.632610 (+ 308us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:20:39.645544 (+ 12934us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 22:20:39.645835 (+ 291us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 22:20:39.646051 6999 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260811 22:20:39.646153 7045 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:39.629326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38547 (local address 127.0.0.1:53066)
0811 22:20:39.629624 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:39.629638 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:39.629760 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:39.630030 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:39.630034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:39.630041 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:20:39.630154 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:39.630160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.630918 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.630921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:39.631920 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:39.631930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.632341 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.632345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:39.632433 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.632462 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:20:39.645756 (+ 13294us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 22:20:39.645957 (+ 201us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.3.126:38547: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260811 22:20:39.646750 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 6879
2026-08-11T22:20:39Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (350 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 22:20:39.699187 29709 mini_oidc.cc:152] Starting JWKS server
I20260811 22:20:39.700045 29709 webserver.cc:460] Webserver started at https://127.0.0.1:41967/ using document root <none> and password file <none>
I20260811 22:20:39.700140 29709 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 22:20:39.700410 29709 webserver.cc:460] Webserver started at http://127.0.0.1:45129/ using document root <none> and password file <none>
2026-08-11T22:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:39Z Disabled control of system clock
I20260811 22:20:39.709865 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35689
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:39587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35689
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41967/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 22:20:39.820453 7059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:39.820741 7059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:39.820798 7059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:39.824856 7059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:39.824923 7059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:39.824946 7059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:39.824966 7059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:39.824987 7059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:20:39.825016 7059 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:20:39.825038 7059 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41967/jwks.json
I20260811 22:20:39.829787 7059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:39587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35689
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41967/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:39.830929 7059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:39.832341 7059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:39.840816 7069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:39.840916 7059 server_base.cc:1034] running on GCE node
W20260811 22:20:39.840831 7071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:39.840816 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:39.841574 7059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:39.842715 7059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:39.843878 7059 hybrid_clock.cc:648] HybridClock initialized: now 1786486839843861 us; error 28 us; skew 500 ppm
I20260811 22:20:39.845619 7059 webserver.cc:460] Webserver started at http://127.29.3.126:41065/ using document root <none> and password file <none>
I20260811 22:20:39.846197 7059 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:39.846266 7059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:39.846501 7059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:39.848462 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "ab82865b6348187742801044f81673b6"
server_key_iv: "934e1856d4e990e9ee300df939f6716b"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.849002 7059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "ab82865b6348187742801044f81673b6"
server_key_iv: "934e1856d4e990e9ee300df939f6716b"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.852531 7059 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 22:20:39.854854 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:39.856047 7059 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:39.856174 7059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed"
format_stamp: "Formatted at 2026-08-11 22:20:39 on dist-test-slave-4tdj"
server_key: "ab82865b6348187742801044f81673b6"
server_key_iv: "934e1856d4e990e9ee300df939f6716b"
server_key_version: "encryptionkey@0"
I20260811 22:20:39.856272 7059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:39.869725 7059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:39.870450 7059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:39.870625 7059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:39.877681 7059 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:35689
I20260811 22:20:39.877697 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:35689 every 8 connection(s)
I20260811 22:20:39.878703 7059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:39.882103 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:39.885000 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 7059
I20260811 22:20:39.885135 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:39.885460 29709 external_mini_cluster.cc:1428] Setting key 81a8ac714962325d68aa3a6ed23c599c
I20260811 22:20:39.888550 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed: Bootstrap starting.
I20260811 22:20:39.890846 7179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:39.891600 7179 log.cc:824] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:39.893579 7179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed: No bootstrap required, opened a new log
I20260811 22:20:39.894702 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:39.886908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.29.3.126:35689)
0811 22:20:39.887581 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:39.887595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:39.887632 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:20:39.888255 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:39.888274 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:39.888687 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:39.889013 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:39.889472 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.890529 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.891441 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.891947 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.892102 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.892926 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:39.892931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:39.892944 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:39.893336 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:39.893392 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:39.893412 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:39.893501 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:39.893648 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 22:20:39.893940 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
I20260811 22:20:39.897733 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:20:39.897583 7179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35689 } }
I20260811 22:20:39.897866 7179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:39.897922 7179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8fd24150b647b7a0f45d6a7f99c3ed, State: Initialized, Role: FOLLOWER
I20260811 22:20:39.898419 7179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35689 } }
I20260811 22:20:39.898587 7179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:39.898665 7179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:39.898775 7179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:39.899852 7179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35689 } }
I20260811 22:20:39.900182 7179 leader_election.cc:302] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c8fd24150b647b7a0f45d6a7f99c3ed; no voters:
I20260811 22:20:39.900503 7179 leader_election.cc:288] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:39.900629 7185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:39.900869 7185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [term 1 LEADER]: Becoming Leader. State: Replica: 2c8fd24150b647b7a0f45d6a7f99c3ed, State: Running, Role: LEADER
I20260811 22:20:39.901199 7185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35689 } }
I20260811 22:20:39.901700 7179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:39.902844 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c8fd24150b647b7a0f45d6a7f99c3ed. Latest consensus state: current_term: 1 leader_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35689 } } }
I20260811 22:20:39.902990 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:39.903478 7206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:39.902840 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8fd24150b647b7a0f45d6a7f99c3ed" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35689 } } }
I20260811 22:20:39.904162 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:39.906806 7206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:39.912041 7206 catalog_manager.cc:1347] Generated new cluster ID: 096c4927f575423eb9a2035b3b843933
I20260811 22:20:39.912108 7206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:39.939566 7206 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:39.940909 7206 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:39.950943 7206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c8fd24150b647b7a0f45d6a7f99c3ed: Generated new TSK 0
I20260811 22:20:39.951884 7206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:20:39.966899 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:39.961704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.29.3.126:35689)
0811 22:20:39.961850 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:39.961855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:39.962029 (+ 174us) server_negotiation.cc:407] Connection header received
0811 22:20:39.962208 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:39.962210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:39.962266 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:39.962399 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:20:39.962648 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.963198 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.964551 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.964788 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.964867 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.965193 (+ 326us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:20:39.966504 (+ 1311us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 22:20:39.966759 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":54}
W20260811 22:20:39.966975 7182 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 22:20:39.969764 7233 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:39.961611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35689 (local address 127.0.0.1:38408)
0811 22:20:39.961935 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:39.961951 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:39.962080 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:39.962407 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:39.962411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:39.962420 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:20:39.962548 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:39.962557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.963307 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.963311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:39.964405 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:39.964419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:39.964905 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:39.964909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:39.964999 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:39.965030 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:20:39.966667 (+ 1637us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 22:20:39.969608 (+ 2941us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.3.126:35689: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41967/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":2737,"thread_start_us":109,"threads_started":1}
I20260811 22:20:39.970381 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 7059
E20260811 22:20:39.994259 7234 webserver.cc:533] Webserver: sslize failed:
E20260811 22:20:39.994377 7234 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-11T22:20:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T22:20:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:40Z Disabled control of system clock
I20260811 22:20:40.035854 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44569
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:38269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:44569
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/testchainca.cert with env {}
W20260811 22:20:40.148388 7240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:40.148674 7240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:40.148732 7240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:40.152830 7240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:40.152897 7240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:40.152920 7240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:40.152940 7240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:40.152961 7240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 22:20:40.152992 7240 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 22:20:40.153013 7240 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 22:20:40.158514 7240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:44569
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:40.159600 7240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:40.160632 7240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:40.169018 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:40.169231 7240 server_base.cc:1034] running on GCE node
W20260811 22:20:40.169072 7245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:40.169046 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:40.169987 7240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:40.171108 7240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:40.172269 7240 hybrid_clock.cc:648] HybridClock initialized: now 1786486840172255 us; error 33 us; skew 500 ppm
I20260811 22:20:40.174074 7240 webserver.cc:460] Webserver started at http://127.29.3.126:39195/ using document root <none> and password file <none>
I20260811 22:20:40.174674 7240 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:40.174739 7240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:40.174925 7240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:40.176926 7240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "fa45cc649eb846339401bb40344d688b"
format_stamp: "Formatted at 2026-08-11 22:20:40 on dist-test-slave-4tdj"
server_key: "036e1c143011034d19cafcffb7916943"
server_key_iv: "91b5f75808c89b7888dd5557d5d5a41b"
server_key_version: "encryptionkey@0"
I20260811 22:20:40.177479 7240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "fa45cc649eb846339401bb40344d688b"
format_stamp: "Formatted at 2026-08-11 22:20:40 on dist-test-slave-4tdj"
server_key: "036e1c143011034d19cafcffb7916943"
server_key_iv: "91b5f75808c89b7888dd5557d5d5a41b"
server_key_version: "encryptionkey@0"
I20260811 22:20:40.181123 7240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:20:40.183545 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:40.184792 7240 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:40.184903 7240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "fa45cc649eb846339401bb40344d688b"
format_stamp: "Formatted at 2026-08-11 22:20:40 on dist-test-slave-4tdj"
server_key: "036e1c143011034d19cafcffb7916943"
server_key_iv: "91b5f75808c89b7888dd5557d5d5a41b"
server_key_version: "encryptionkey@0"
I20260811 22:20:40.185016 7240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:40.211465 7240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:40.212288 7240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:40.212491 7240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:40.219818 7240 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:44569
I20260811 22:20:40.219846 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:44569 every 8 connection(s)
I20260811 22:20:40.220791 7240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:40.221848 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 7240
I20260811 22:20:40.221973 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:40.222330 29709 external_mini_cluster.cc:1428] Setting key 2944363e1a3b296733e0d6d59dbb4369
I20260811 22:20:40.224203 7374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:40.230690 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b: Bootstrap starting.
I20260811 22:20:40.231482 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:40.223676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.29.3.126:44569)
0811 22:20:40.224346 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:40.224368 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:40.224407 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:20:40.225042 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:40.225059 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:40.225469 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:40.225772 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:40.226110 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:40.227123 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:40.228285 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:40.228742 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:40.228853 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:40.229723 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:40.229728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:40.229740 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:20:40.230128 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:40.230181 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:20:40.230219 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:40.230326 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:40.230480 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 22:20:40.230751 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260811 22:20:40.233143 7374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:40.233982 7374 log.cc:824] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:40.234349 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:20:40.236047 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b: No bootstrap required, opened a new log
I20260811 22:20:40.239598 7374 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa45cc649eb846339401bb40344d688b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44569 } }
I20260811 22:20:40.239830 7374 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:40.239871 7374 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa45cc649eb846339401bb40344d688b, State: Initialized, Role: FOLLOWER
I20260811 22:20:40.240235 7374 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa45cc649eb846339401bb40344d688b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44569 } }
I20260811 22:20:40.240346 7374 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:40.240391 7374 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:40.240460 7374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:40.241315 7374 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa45cc649eb846339401bb40344d688b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44569 } }
I20260811 22:20:40.241618 7374 leader_election.cc:302] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa45cc649eb846339401bb40344d688b; no voters:
I20260811 22:20:40.241887 7374 leader_election.cc:288] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:40.241994 7380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:40.242210 7380 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [term 1 LEADER]: Becoming Leader. State: Replica: fa45cc649eb846339401bb40344d688b, State: Running, Role: LEADER
I20260811 22:20:40.242583 7380 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa45cc649eb846339401bb40344d688b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44569 } }
I20260811 22:20:40.243109 7374 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:40.244181 7382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa45cc649eb846339401bb40344d688b. Latest consensus state: current_term: 1 leader_uuid: "fa45cc649eb846339401bb40344d688b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa45cc649eb846339401bb40344d688b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44569 } } }
I20260811 22:20:40.244325 7382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:40.244611 7381 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa45cc649eb846339401bb40344d688b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa45cc649eb846339401bb40344d688b" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44569 } } }
I20260811 22:20:40.244717 7395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:40.244784 7381 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:40.248045 7395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:40.253608 7395 catalog_manager.cc:1347] Generated new cluster ID: c7333805b2024c0b83242f9a2eff60f1
I20260811 22:20:40.253723 7395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:40.279685 7395 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:40.280642 7395 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:40.289533 7395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa45cc649eb846339401bb40344d688b: Generated new TSK 0
I20260811 22:20:40.290277 7395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:20:40.318420 7378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:40.304838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52956 (local address 127.29.3.126:44569)
0811 22:20:40.304956 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:40.304961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:40.305165 (+ 204us) server_negotiation.cc:407] Connection header received
0811 22:20:40.305349 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:40.305351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:40.305401 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:40.305497 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0811 22:20:40.305738 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:40.306264 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:40.307130 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:40.307347 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:40.307435 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:40.307757 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 22:20:40.317939 (+ 10182us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 22:20:40.318242 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52956: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":41}
W20260811 22:20:40.318508 7378 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52956: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 22:20:40.318524 7423 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:40.304793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44569 (local address 127.0.0.1:52956)
0811 22:20:40.305082 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:40.305093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:40.305202 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:40.305508 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:40.305511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:40.305519 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0811 22:20:40.305629 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:40.305636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:40.306382 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:40.306386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:40.307011 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:40.307019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:40.307470 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:40.307475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:40.307565 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:40.307587 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 22:20:40.318139 (+ 10552us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 22:20:40.318351 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.3.126:44569: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":110,"threads_started":1}
I20260811 22:20:40.319344 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 7240
2026-08-11T22:20:40Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (323 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (127 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (126 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:40 dist-test-slave-4tdj krb5kdc[7453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:40 dist-test-slave-4tdj krb5kdc[7453](info): set up 2 sockets
Aug 11 22:20:40 dist-test-slave-4tdj krb5kdc[7453](info): commencing operation
krb5kdc: starting...
W20260811 22:20:41.774137 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:41 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:41.897421 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38419
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:38419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:42.009902 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:42.010187 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:42.010245 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:42.014218 7472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:42.014416 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:42.014485 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:42.014513 7472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:42.014542 7472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:42.020430 7472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:38419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:42.021564 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:42.022622 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:42.031337 7484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.031505 7472 server_base.cc:1034] running on GCE node
W20260811 22:20:42.031337 7480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:42.031419 7481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.032176 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:42.033278 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:42.034438 7472 hybrid_clock.cc:648] HybridClock initialized: now 1786486842034416 us; error 50 us; skew 500 ppm
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.037168 7472 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:42.038077 7472 webserver.cc:460] Webserver started at http://127.29.3.126:41451/ using document root <none> and password file <none>
I20260811 22:20:42.038667 7472 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:42.038756 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:42.038968 7472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:42.041018 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "3d113f3537784318973843e55996a7f0"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "00dc93867c1fdc6c87ff7fe9ac8a4a6e"
server_key_iv: "631599c9b54edba09c8ccec4d4672c47"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.041584 7472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "3d113f3537784318973843e55996a7f0"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "00dc93867c1fdc6c87ff7fe9ac8a4a6e"
server_key_iv: "631599c9b54edba09c8ccec4d4672c47"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.045351 7472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:20:42.047710 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.048859 7472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:42.048976 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "3d113f3537784318973843e55996a7f0"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "00dc93867c1fdc6c87ff7fe9ac8a4a6e"
server_key_iv: "631599c9b54edba09c8ccec4d4672c47"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.049072 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:42.060060 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:42.063068 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:42.063242 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:42.070006 7472 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:38419
I20260811 22:20:42.070020 7573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:38419 every 8 connection(s)
I20260811 22:20:42.070950 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:42.073664 7574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:42.074229 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 7472
I20260811 22:20:42.074378 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:42.074723 29709 external_mini_cluster.cc:1428] Setting key 2af6b9ac5635f646add555c386a06044
I20260811 22:20:42.079874 7574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0: Bootstrap starting.
I20260811 22:20:42.082588 7574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:42.083401 7574 log.cc:824] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:42.085485 7574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0: No bootstrap required, opened a new log
I20260811 22:20:42.088321 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.076061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53084 (local address 127.29.3.126:38419)
0811 22:20:42.076461 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.076477 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.076512 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:20:42.077146 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.077163 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.077494 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.077768 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.078675 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.079644 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.080755 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.081221 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.081345 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.084070 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.084076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.084093 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.084124 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.085672 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.086120 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.086124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.086129 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.086243 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.086529 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.086531 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.086532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.086907 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.087045 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.087335 (+ 290us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.087604 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
I20260811 22:20:42.089428 7574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d113f3537784318973843e55996a7f0" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38419 } }
I20260811 22:20:42.089697 7574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:42.089768 7574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d113f3537784318973843e55996a7f0, State: Initialized, Role: FOLLOWER
I20260811 22:20:42.090232 7574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d113f3537784318973843e55996a7f0" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38419 } }
I20260811 22:20:42.090412 7574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:42.090524 7574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:42.090632 7574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:42.091540 7574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d113f3537784318973843e55996a7f0" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38419 } }
I20260811 22:20:42.091867 7574 leader_election.cc:302] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d113f3537784318973843e55996a7f0; no voters:
I20260811 22:20:42.092146 7574 leader_election.cc:288] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:42.092275 7584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:42.092489 7584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [term 1 LEADER]: Becoming Leader. State: Replica: 3d113f3537784318973843e55996a7f0, State: Running, Role: LEADER
I20260811 22:20:42.092834 7584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d113f3537784318973843e55996a7f0" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38419 } }
I20260811 22:20:42.093307 7574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:42.094360 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d113f3537784318973843e55996a7f0. Latest consensus state: current_term: 1 leader_uuid: "3d113f3537784318973843e55996a7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d113f3537784318973843e55996a7f0" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38419 } } }
I20260811 22:20:42.094523 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:42.094846 7585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d113f3537784318973843e55996a7f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d113f3537784318973843e55996a7f0" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38419 } } }
I20260811 22:20:42.095000 7585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:42.095188 7599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:42.097937 7599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:42.103515 7599 catalog_manager.cc:1347] Generated new cluster ID: 95a00f0d80274b9fb5b39f06af00265f
I20260811 22:20:42.103579 7599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:42.113107 7599 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:42.114037 7599 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:42.119414 7599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d113f3537784318973843e55996a7f0: Generated new TSK 0
I20260811 22:20:42.120190 7599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:42.149881 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38419
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:42.262930 7616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:42.263182 7616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:42.263230 7616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:42.267287 7616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:42.267345 7616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:42.267419 7616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:42.272624 7616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:42.273658 7616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:42.274734 7616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:42.284221 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.284363 7616 server_base.cc:1034] running on GCE node
W20260811 22:20:42.284271 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:42.284286 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.284967 7616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:42.285586 7616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:42.286752 7616 hybrid_clock.cc:648] HybridClock initialized: now 1786486842286732 us; error 31 us; skew 500 ppm
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.292914 7616 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:42.293967 7616 webserver.cc:460] Webserver started at http://127.29.3.65:32835/ using document root <none> and password file <none>
I20260811 22:20:42.294633 7616 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:42.294704 7616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:42.294917 7616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:42.297061 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "d0f948b5a8214187b5cbf54b80aa629b"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "e6beb8a0e5e5a22c6e1d9d115ec756f2"
server_key_iv: "0371de2169717535634d81a602f7e07a"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.297623 7616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "d0f948b5a8214187b5cbf54b80aa629b"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "e6beb8a0e5e5a22c6e1d9d115ec756f2"
server_key_iv: "0371de2169717535634d81a602f7e07a"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.301424 7616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:20:42.303958 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.305190 7616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:42.305320 7616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "d0f948b5a8214187b5cbf54b80aa629b"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "e6beb8a0e5e5a22c6e1d9d115ec756f2"
server_key_iv: "0371de2169717535634d81a602f7e07a"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.305428 7616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:42.318800 7616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:42.321877 7616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:42.322064 7616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:42.322708 7616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:42.323568 7616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:42.323621 7616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.323674 7616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:42.323698 7616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.332506 7616 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:33983
I20260811 22:20:42.332551 7854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:33983 every 8 connection(s)
I20260811 22:20:42.333500 7616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:42.337267 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 7616
I20260811 22:20:42.337392 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:42.337695 29709 external_mini_cluster.cc:1428] Setting key cc94928acfcf88064437b73b74ed7cd8
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:42.347720 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.335295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:52075 (local address 127.29.3.126:38419)
0811 22:20:42.335417 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.335421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.336269 (+ 848us) server_negotiation.cc:407] Connection header received
0811 22:20:42.337634 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.337638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.337700 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.337784 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.339260 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.339787 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.340917 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.341212 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.341300 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.344025 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.344028 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.344031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.344057 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.345446 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.346076 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.346079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.346080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.346150 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.346572 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.346574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.346575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.346763 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.346838 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.347482 (+ 644us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.347590 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:20:42.348939 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.335638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38419 (local address 127.29.3.65:52075)
0811 22:20:42.336097 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:42.336142 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:42.337390 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:42.338024 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:42.338032 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:42.338542 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:42.339111 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:42.339132 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.339917 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.339920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:42.340801 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:42.340811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.341367 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.341371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:42.341566 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.342169 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:42.342191 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:42.343827 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:42.345577 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:42.345584 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:42.345602 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:42.345983 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:42.346248 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:42.346251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:42.346253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:42.346475 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:42.346865 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:42.346872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:42.347225 (+ 353us) client_negotiation.cc:769] Sending connection context
0811 22:20:42.347451 (+ 226us) client_negotiation.cc:240] Negotiation successful
0811 22:20:42.347827 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":114,"threads_started":1}
I20260811 22:20:42.350241 7855 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38419
I20260811 22:20:42.350512 7855 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:42.351015 7855 heartbeater.cc:511] Master 127.29.3.126:38419 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:20:42.352586 7523 ts_manager.cc:194] Registered new tserver with Master: d0f948b5a8214187b5cbf54b80aa629b (127.29.3.65:33983)
I20260811 22:20:42.354022 7523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:52075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:42.366988 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38419
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:42.478901 7862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:42.479162 7862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:42.479204 7862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:42.483220 7862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:42.483274 7862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:42.483332 7862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:42.488617 7862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:42.489605 7862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:42.490597 7862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:42.500181 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:42.500236 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.500367 7862 server_base.cc:1034] running on GCE node
W20260811 22:20:42.500200 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.500876 7862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:42.501525 7862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:42.502682 7862 hybrid_clock.cc:648] HybridClock initialized: now 1786486842502664 us; error 28 us; skew 500 ppm
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.505352 7862 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:42.506201 7862 webserver.cc:460] Webserver started at http://127.29.3.66:40293/ using document root <none> and password file <none>
I20260811 22:20:42.506776 7862 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:42.506848 7862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:42.507058 7862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:42.509142 7862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "722e6dbbc3a14c62bbce72c0178f904f"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "73673f3d918ce235f6b480d928221274"
server_key_iv: "0306e5dfa1a19356d0bc620bb1a22812"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.509714 7862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "722e6dbbc3a14c62bbce72c0178f904f"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "73673f3d918ce235f6b480d928221274"
server_key_iv: "0306e5dfa1a19356d0bc620bb1a22812"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.513355 7862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:20:42.515753 7895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.516886 7862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:42.516995 7862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "722e6dbbc3a14c62bbce72c0178f904f"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "73673f3d918ce235f6b480d928221274"
server_key_iv: "0306e5dfa1a19356d0bc620bb1a22812"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.517089 7862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:42.544160 7862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:42.547714 7862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:42.547904 7862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:42.548521 7862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:42.549366 7862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:42.549544 7862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.549639 7862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:42.549674 7862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.558696 7862 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:41809
I20260811 22:20:42.558779 8118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:41809 every 8 connection(s)
I20260811 22:20:42.559593 7862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:42.563968 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 7862
I20260811 22:20:42.564078 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:42.564380 29709 external_mini_cluster.cc:1428] Setting key 594d1517bba6c81fdc9eaaf30208385e
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:42.575214 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.561478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:53557 (local address 127.29.3.126:38419)
0811 22:20:42.561640 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.561644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.562465 (+ 821us) server_negotiation.cc:407] Connection header received
0811 22:20:42.563834 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.563837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.563899 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.564116 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.565564 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.566179 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.567713 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.567996 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.568117 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.570671 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.570675 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.570678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.570708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.572497 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.573159 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.573163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.573165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.573262 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.573719 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.573724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.573726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.573982 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.574102 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.574890 (+ 788us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.575027 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:20:42.576272 8122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.561819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38419 (local address 127.29.3.66:53557)
0811 22:20:42.562275 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:42.562356 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:42.563605 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:42.564333 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:42.564342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:42.564842 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:42.565421 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:42.565436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.566340 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.566343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:42.567563 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:42.567575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.568158 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.568161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:42.568305 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.568903 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:42.568919 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:42.570481 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:42.572637 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:42.572644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:42.572660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:42.573036 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:42.573381 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:42.573384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:42.573386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:42.573567 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:42.574119 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:42.574128 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:42.574575 (+ 447us) client_negotiation.cc:769] Sending connection context
0811 22:20:42.574781 (+ 206us) client_negotiation.cc:240] Negotiation successful
0811 22:20:42.575173 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260811 22:20:42.577625 8120 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38419
I20260811 22:20:42.577908 8120 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
I20260811 22:20:42.578511 8120 heartbeater.cc:511] Master 127.29.3.126:38419 requested a full tablet report, sending...
I20260811 22:20:42.579828 7523 ts_manager.cc:194] Registered new tserver with Master: 722e6dbbc3a14c62bbce72c0178f904f (127.29.3.66:41809)
I20260811 22:20:42.580789 7523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:53557
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:42.598733 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38419
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:42.715535 8125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:42.715802 8125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:42.715862 8125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:42.719861 8125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:42.719929 8125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:42.720009 8125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:42.725785 8125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:42.726868 8125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:42.727852 8125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:42.737425 8147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:42.737460 8144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:42.737459 8135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:42.737718 8125 server_base.cc:1034] running on GCE node
I20260811 22:20:42.738185 8125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:42.738818 8125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:42.739990 8125 hybrid_clock.cc:648] HybridClock initialized: now 1786486842739961 us; error 41 us; skew 500 ppm
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.742915 8125 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:42.743764 8125 webserver.cc:460] Webserver started at http://127.29.3.67:38419/ using document root <none> and password file <none>
I20260811 22:20:42.744290 8125 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:42.744350 8125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:42.744565 8125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:42.746609 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "30e3401f1c274583a52fd12aa9984c93"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "b7cc8bdbeee29889fda3d7af2cc411b0"
server_key_iv: "76ba36988083df8c6234bfb70d194e24"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.747135 8125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "30e3401f1c274583a52fd12aa9984c93"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "b7cc8bdbeee29889fda3d7af2cc411b0"
server_key_iv: "76ba36988083df8c6234bfb70d194e24"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.750746 8125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:42.753096 8157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.754343 8125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:42.754451 8125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "30e3401f1c274583a52fd12aa9984c93"
format_stamp: "Formatted at 2026-08-11 22:20:42 on dist-test-slave-4tdj"
server_key: "b7cc8bdbeee29889fda3d7af2cc411b0"
server_key_iv: "76ba36988083df8c6234bfb70d194e24"
server_key_version: "encryptionkey@0"
I20260811 22:20:42.754550 8125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:42.775923 8125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:42.778690 8125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:42.778851 8125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:42.779440 8125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:42.780268 8125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:42.780313 8125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.780354 8125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:42.780385 8125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:42.789230 8125 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:36497
I20260811 22:20:42.789383 8356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:36497 every 8 connection(s)
I20260811 22:20:42.790139 8125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:42.795354 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 8125
I20260811 22:20:42.795447 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:42.795698 29709 external_mini_cluster.cc:1428] Setting key 9de6a1f1c4c8b2a3d789fd8506ee3b9a
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:42.803323 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.792061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:58045 (local address 127.29.3.126:38419)
0811 22:20:42.792181 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.792185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.793069 (+ 884us) server_negotiation.cc:407] Connection header received
0811 22:20:42.794512 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.794515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.794568 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.794675 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.795942 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.796483 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.797598 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.797828 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.797915 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.800231 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.800235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.800239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.800264 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.801328 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.801804 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.801808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.801810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.801888 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.802207 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.802211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.802213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.802438 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.802528 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.803019 (+ 491us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.803149 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 22:20:42.804121 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.792383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38419 (local address 127.29.3.67:58045)
0811 22:20:42.792896 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:42.792940 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:42.794274 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:42.794836 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:42.794843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:42.795252 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:42.795798 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:42.795814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.796605 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.796607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:42.797481 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:42.797490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.797963 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.797967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:42.798133 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.798693 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:42.798710 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:42.800067 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:42.801435 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:42.801441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:42.801457 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:42.801711 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:42.801973 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:42.801976 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:42.801978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:42.802112 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:42.802542 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:42.802548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:42.802791 (+ 243us) client_negotiation.cc:769] Sending connection context
0811 22:20:42.802960 (+ 169us) client_negotiation.cc:240] Negotiation successful
0811 22:20:42.803285 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":144,"threads_started":1}
I20260811 22:20:42.805210 8358 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38419
I20260811 22:20:42.805455 8358 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:42.805948 8358 heartbeater.cc:511] Master 127.29.3.126:38419 requested a full tablet report, sending...
I20260811 22:20:42.806990 7523 ts_manager.cc:194] Registered new tserver with Master: 30e3401f1c274583a52fd12aa9984c93 (127.29.3.67:36497)
I20260811 22:20:42.807540 7523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:58045
I20260811 22:20:42.809331 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/test-admin.keytab.
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.826069 29709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 22:20:42.828799 8370 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:42.827420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43086 (local address 127.29.3.65:33983)
0811 22:20:42.827722 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.827736 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.827760 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:20:42.827810 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.827815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.827996 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.828114 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.828625 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43086: BlockingRecv error: recv got EOF from 127.0.0.1:43086 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":109,"threads_started":1}
W20260811 22:20:42.828867 29709 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc' of type FILE
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.829885 29709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.830797 29709 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:42.839569 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.831605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.29.3.65:33983)
0811 22:20:42.831738 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.831742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.831812 (+ 70us) server_negotiation.cc:407] Connection header received
0811 22:20:42.831901 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.831904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.831942 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.832007 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.832490 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.833292 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.834079 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.834499 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.834582 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.836590 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.836599 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.836612 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.836636 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.838076 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.838455 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.838461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.838462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.838540 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.838723 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.838725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.838726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.838947 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.839102 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.839259 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.839407 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 22:20:42.843182 8370 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:42.842211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43106 (local address 127.29.3.65:33983)
0811 22:20:42.842351 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.842355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.842489 (+ 134us) server_negotiation.cc:407] Connection header received
0811 22:20:42.842594 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.842596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.842634 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.842692 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.843092 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43106: BlockingRecv error: recv got EOF from 127.0.0.1:43106 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260811 22:20:42.843312 29709 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc' of type FILE
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.844265 29709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.845065 29709 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:42.853075 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.845994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.29.3.65:33983)
0811 22:20:42.846109 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.846113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.846126 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:42.846165 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.846167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.846207 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.846269 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.846766 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.847260 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.848011 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.848230 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.848315 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.850387 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.850390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.850392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.850413 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.851544 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.851930 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.851934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.851935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.852046 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.852319 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.852322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.852324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.852488 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.852599 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.852829 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.852932 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260811 22:20:42.855552 8370 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:42.854659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.29.3.65:33983)
0811 22:20:42.854783 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.854786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.854888 (+ 102us) server_negotiation.cc:407] Connection header received
0811 22:20:42.855003 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.855005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.855039 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.855097 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.855470 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43124: BlockingRecv error: recv got EOF from 127.0.0.1:43124 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 22:20:42.855717 29709 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786486810933177-29709-0/krb5kdc/krb5cc' of type FILE
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.856606 29709 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:42.857395 29709 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 22:20:42 dist-test-slave-4tdj krb5kdc[7453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:20:42.865312 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:42.858125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.29.3.65:33983)
0811 22:20:42.858266 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:42.858270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:42.858401 (+ 131us) server_negotiation.cc:407] Connection header received
0811 22:20:42.858486 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:42.858488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:42.858525 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:42.858585 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:42.858998 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.859528 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.860259 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:42.860464 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:42.860539 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:42.862635 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:42.862638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:42.862640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:42.862674 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:42.863900 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.864276 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.864279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.864280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.864345 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:42.864559 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:42.864561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:42.864562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:42.864724 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:42.864816 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:42.865061 (+ 245us) server_negotiation.cc:299] Negotiation successful
0811 22:20:42.865147 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:20:42.866254 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 7616
I20260811 22:20:42.899564 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 7862
I20260811 22:20:42.918784 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 8125
I20260811 22:20:42.937674 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 7472
2026-08-11T22:20:42Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2361 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:43 dist-test-slave-4tdj krb5kdc[8389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:43 dist-test-slave-4tdj krb5kdc[8389](info): set up 2 sockets
Aug 11 22:20:43 dist-test-slave-4tdj krb5kdc[8389](info): commencing operation
krb5kdc: starting...
W20260811 22:20:44.110077 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab.
Entry for principal oryx/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab.
I20260811 22:20:44.235345 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39659
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab
--principal=oryx/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:44.351377 8408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:44.351634 8408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:44.351681 8408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:44.355574 8408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:44.355628 8408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:44.355645 8408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:44.355679 8408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:44.355692 8408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:44.360826 8408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab
--principal=oryx/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:44.361785 8408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:44.362780 8408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:44.371253 8418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:44.371415 8408 server_base.cc:1034] running on GCE node
W20260811 22:20:44.371218 8417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:44.371268 8420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:44.372109 8408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:44.373152 8408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:44.374333 8408 hybrid_clock.cc:648] HybridClock initialized: now 1786486844374308 us; error 38 us; skew 500 ppm
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:44.377067 8408 init.cc:377] Logged in from keytab as oryx/127.29.3.126@KRBTEST.COM (short username oryx)
I20260811 22:20:44.377954 8408 webserver.cc:460] Webserver started at http://127.29.3.126:35395/ using document root <none> and password file <none>
I20260811 22:20:44.378523 8408 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:44.378587 8408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:44.378773 8408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:44.380781 8408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "4920be7096e146b48b848d87c4371cdb"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "ec61ce0a46f6080a6e66c70f57755ef3"
server_key_iv: "0e69b8a50594a2ecf37c6ea534d3a8f6"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.381302 8408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "4920be7096e146b48b848d87c4371cdb"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "ec61ce0a46f6080a6e66c70f57755ef3"
server_key_iv: "0e69b8a50594a2ecf37c6ea534d3a8f6"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.385001 8408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:20:44.387419 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.388691 8408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:44.388810 8408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "4920be7096e146b48b848d87c4371cdb"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "ec61ce0a46f6080a6e66c70f57755ef3"
server_key_iv: "0e69b8a50594a2ecf37c6ea534d3a8f6"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.388906 8408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:44.401398 8408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:44.404433 8408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:44.404613 8408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:44.411509 8408 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:39659
I20260811 22:20:44.411789 8523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:39659 every 8 connection(s)
I20260811 22:20:44.412478 8408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:44.415233 8524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:44.420833 8524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Bootstrap starting.
I20260811 22:20:44.421659 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 8408
I20260811 22:20:44.421789 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:44.422049 29709 external_mini_cluster.cc:1428] Setting key c64be4206cdc2220444ced257d5f74d9
I20260811 22:20:44.423585 8524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:44.424315 8524 log.cc:824] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:44.426558 8524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: No bootstrap required, opened a new log
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:44.430866 8524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } }
I20260811 22:20:44.431120 8524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:44.431181 8524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4920be7096e146b48b848d87c4371cdb, State: Initialized, Role: FOLLOWER
I20260811 22:20:44.431689 8524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } }
I20260811 22:20:44.431856 8524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:44.431941 8524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:44.432041 8524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:44.432901 8524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } }
I20260811 22:20:44.433235 8524 leader_election.cc:302] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4920be7096e146b48b848d87c4371cdb; no voters:
I20260811 22:20:44.433580 8524 leader_election.cc:288] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:44.433682 8535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:44.433928 8535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 LEADER]: Becoming Leader. State: Replica: 4920be7096e146b48b848d87c4371cdb, State: Running, Role: LEADER
I20260811 22:20:44.434335 8535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } }
I20260811 22:20:44.434993 8524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:44.436116 8537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } }
I20260811 22:20:44.436232 8537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:44.436537 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:44.423184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46140 (local address 127.29.3.126:39659)
0811 22:20:44.423594 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:44.423610 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:44.423647 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:20:44.424333 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:44.424350 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:44.424748 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:44.425119 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:44.425978 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.427026 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.428149 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.428603 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.428724 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:44.431427 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:44.431433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:44.431449 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:44.431481 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:44.433206 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:44.433679 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:44.433685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:44.433692 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:44.433844 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:44.434184 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:44.434188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:44.434190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:44.434731 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:44.434954 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:44.435220 (+ 266us) server_negotiation.cc:299] Negotiation successful
0811 22:20:44.435530 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":116,"threads_started":1}
I20260811 22:20:44.436667 8538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4920be7096e146b48b848d87c4371cdb. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } }
I20260811 22:20:44.436743 8538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:44.437000 8550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:44.439734 8550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:44.445098 8550 catalog_manager.cc:1347] Generated new cluster ID: d1e47123fc144161a2424f9e06620014
I20260811 22:20:44.445166 8550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:44.467798 8550 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:44.468747 8550 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:44.477535 8550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Generated new TSK 0
I20260811 22:20:44.478355 8550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab.
Entry for principal oryx/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab.
I20260811 22:20:44.515100 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39659
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:44.628273 8567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:44.628559 8567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:44.628618 8567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:44.632652 8567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:44.632720 8567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:44.632802 8567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:44.638248 8567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:44.639420 8567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:44.640472 8567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:44.650238 8575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:44.650411 8567 server_base.cc:1034] running on GCE node
W20260811 22:20:44.650278 8576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:44.650233 8579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:44.650944 8567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:44.651595 8567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:44.652745 8567 hybrid_clock.cc:648] HybridClock initialized: now 1786486844652718 us; error 42 us; skew 500 ppm
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:44.655509 8567 init.cc:377] Logged in from keytab as oryx/127.29.3.65@KRBTEST.COM (short username oryx)
I20260811 22:20:44.656495 8567 webserver.cc:460] Webserver started at http://127.29.3.65:38475/ using document root <none> and password file <none>
I20260811 22:20:44.657037 8567 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:44.657095 8567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:44.657275 8567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:44.659926 8567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "aed735695b25402aa239b0452e6ce7e6"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "5a4ae9a394f149900246105b85ab85e2"
server_key_iv: "0c1ac9fabbea0d6c43f918ea3b7d81cf"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.660598 8567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "aed735695b25402aa239b0452e6ce7e6"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "5a4ae9a394f149900246105b85ab85e2"
server_key_iv: "0c1ac9fabbea0d6c43f918ea3b7d81cf"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.664387 8567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:44.666855 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.668083 8567 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:44.668198 8567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "aed735695b25402aa239b0452e6ce7e6"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "5a4ae9a394f149900246105b85ab85e2"
server_key_iv: "0c1ac9fabbea0d6c43f918ea3b7d81cf"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.668300 8567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:44.683537 8567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:44.686416 8567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:44.686604 8567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:44.687204 8567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:44.688066 8567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:44.688115 8567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.688170 8567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:44.688200 8567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.697037 8567 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:33273
I20260811 22:20:44.697067 8781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:33273 every 8 connection(s)
I20260811 22:20:44.697928 8567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:44.702108 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 8567
I20260811 22:20:44.702263 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:44.702620 29709 external_mini_cluster.cc:1428] Setting key 7060c389bedb63ba286c3a71af81afc8
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:44.712559 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:44.699766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38685 (local address 127.29.3.126:39659)
0811 22:20:44.699912 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:44.699916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:44.700739 (+ 823us) server_negotiation.cc:407] Connection header received
0811 22:20:44.702441 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:44.702445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:44.702539 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:44.702619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:44.704039 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.704746 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.705912 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.706223 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.706335 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:44.708932 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:44.708937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:44.708939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:44.708972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:44.710524 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:44.711022 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:44.711026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:44.711028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:44.711122 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:44.711437 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:44.711440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:44.711442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:44.711680 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:44.711788 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:44.712252 (+ 464us) server_negotiation.cc:299] Negotiation successful
0811 22:20:44.712383 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for oryx/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:20:44.713331 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:44.700131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:38685)
0811 22:20:44.700581 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:44.700634 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:44.702136 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:44.702845 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:44.702855 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:44.703251 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:44.703896 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:44.703915 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.704882 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.704885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:44.705787 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:44.705798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.706397 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.706401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:44.706606 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:44.707184 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:44.707205 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:44.708755 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:44.710635 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:44.710640 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:44.710653 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:44.710914 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:44.711205 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:44.711208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:44.711218 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:44.711352 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:44.711772 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:44.711778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:44.712013 (+ 235us) client_negotiation.cc:769] Sending connection context
0811 22:20:44.712171 (+ 158us) client_negotiation.cc:240] Negotiation successful
0811 22:20:44.712439 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260811 22:20:44.714677 8783 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39659
I20260811 22:20:44.714934 8783 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:44.715540 8783 heartbeater.cc:511] Master 127.29.3.126:39659 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.66@KRBTEST.COM" created.
I20260811 22:20:44.717341 8464 ts_manager.cc:194] Registered new tserver with Master: aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:44.718683 8464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:38685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab.
Entry for principal oryx/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab.
I20260811 22:20:44.733059 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39659
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:44.846757 8791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:44.847023 8791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:44.847087 8791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:44.851094 8791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:44.851157 8791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:44.851246 8791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:44.856607 8791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:44.857735 8791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:44.858803 8791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:44.868563 8799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:44.868587 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:44.868563 8803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:44.868875 8791 server_base.cc:1034] running on GCE node
I20260811 22:20:44.869277 8791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:44.869933 8791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:44.871096 8791 hybrid_clock.cc:648] HybridClock initialized: now 1786486844871062 us; error 49 us; skew 500 ppm
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:44.873808 8791 init.cc:377] Logged in from keytab as oryx/127.29.3.66@KRBTEST.COM (short username oryx)
I20260811 22:20:44.874770 8791 webserver.cc:460] Webserver started at http://127.29.3.66:37329/ using document root <none> and password file <none>
I20260811 22:20:44.875326 8791 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:44.875396 8791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:44.875592 8791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:44.877645 8791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "345fc1b58590433a8cb2eca25e4deec7"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "d20fff3c28d60675678bb9dab1e60fa7"
server_key_iv: "84235fc499711116f41bbf2582df2a0b"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.878201 8791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "345fc1b58590433a8cb2eca25e4deec7"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "d20fff3c28d60675678bb9dab1e60fa7"
server_key_iv: "84235fc499711116f41bbf2582df2a0b"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.881956 8791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:44.884490 8818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.885774 8791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:44.885895 8791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "345fc1b58590433a8cb2eca25e4deec7"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "d20fff3c28d60675678bb9dab1e60fa7"
server_key_iv: "84235fc499711116f41bbf2582df2a0b"
server_key_version: "encryptionkey@0"
I20260811 22:20:44.886006 8791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:44.917147 8791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:44.920130 8791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:44.920324 8791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:44.920958 8791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:44.921854 8791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:44.921904 8791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.921960 8791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:44.921990 8791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:44.930832 8791 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:46393
I20260811 22:20:44.930862 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:46393 every 8 connection(s)
I20260811 22:20:44.931762 8791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:44.940356 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 8791
I20260811 22:20:44.940481 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:44.940752 29709 external_mini_cluster.cc:1428] Setting key f825d51602fc2c5f4da193f09bcc258d
Aug 11 22:20:44 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:44.945819 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:44.933567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:38455 (local address 127.29.3.126:39659)
0811 22:20:44.933747 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:44.933752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:44.934618 (+ 866us) server_negotiation.cc:407] Connection header received
0811 22:20:44.936135 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:44.936138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:44.936201 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:44.936280 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:44.937590 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.938110 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.939234 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.939451 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.939534 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:44.942005 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:44.942008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:44.942010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:44.942037 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:44.943541 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:44.944067 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:44.944071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:44.944072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:44.944176 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:44.944577 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:44.944580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:44.944581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:44.944773 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:44.944851 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:44.945560 (+ 709us) server_negotiation.cc:299] Negotiation successful
0811 22:20:44.945690 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:20:44.946949 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:44.933960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:38455)
0811 22:20:44.934462 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:44.934507 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:44.935905 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:44.936435 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:44.936442 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:44.936861 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:44.937442 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:44.937460 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.938208 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.938211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:44.939123 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:44.939133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:44.939570 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:44.939573 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:44.939720 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:44.940280 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:44.940296 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:44.941833 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:44.943661 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:44.943666 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:44.943680 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:44.943966 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:44.944270 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:44.944274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:44.944276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:44.944483 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:44.944870 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:44.944877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:44.945238 (+ 361us) client_negotiation.cc:769] Sending connection context
0811 22:20:44.945470 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 22:20:44.945854 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
I20260811 22:20:44.948292 9016 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39659
I20260811 22:20:44.948593 9016 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:44.949276 9016 heartbeater.cc:511] Master 127.29.3.126:39659 requested a full tablet report, sending...
I20260811 22:20:44.950603 8464 ts_manager.cc:194] Registered new tserver with Master: 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66:46393)
I20260811 22:20:44.951428 8464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:38455
WARNING: no policy specified for oryx/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab.
Entry for principal oryx/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab.
I20260811 22:20:44.973129 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39659
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:45.087958 9024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:45.088234 9024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:45.088277 9024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:45.092407 9024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:45.092476 9024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:45.092542 9024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:45.097998 9024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:45.099222 9024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:45.100288 9024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:45.110100 9037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.110200 9040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.110100 9036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.110378 9024 server_base.cc:1034] running on GCE node
I20260811 22:20:45.110854 9024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:45.111498 9024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:45.112654 9024 hybrid_clock.cc:648] HybridClock initialized: now 1786486845112641 us; error 27 us; skew 500 ppm
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:45.115538 9024 init.cc:377] Logged in from keytab as oryx/127.29.3.67@KRBTEST.COM (short username oryx)
I20260811 22:20:45.116437 9024 webserver.cc:460] Webserver started at http://127.29.3.67:39385/ using document root <none> and password file <none>
I20260811 22:20:45.116997 9024 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.117065 9024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.117266 9024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:45.119365 9024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "d8dbab3c846c4825b4f0315139d4c099"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "e58759585696d1cb682d0ed35f14bd3d"
server_key_iv: "76e68b64f5bdea3d851be90257a68b18"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.119911 9024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "d8dbab3c846c4825b4f0315139d4c099"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "e58759585696d1cb682d0ed35f14bd3d"
server_key_iv: "76e68b64f5bdea3d851be90257a68b18"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.123555 9024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:45.125947 9056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.127247 9024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:45.127348 9024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "d8dbab3c846c4825b4f0315139d4c099"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "e58759585696d1cb682d0ed35f14bd3d"
server_key_iv: "76e68b64f5bdea3d851be90257a68b18"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.127435 9024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.152341 9024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:45.155664 9024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:45.155871 9024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:45.156488 9024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:45.157404 9024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:45.157455 9024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.157497 9024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:45.157517 9024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.166769 9024 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:40707
I20260811 22:20:45.166795 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:40707 every 8 connection(s)
I20260811 22:20:45.167722 9024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:45.170667 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 9024
I20260811 22:20:45.170825 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:45.171205 29709 external_mini_cluster.cc:1428] Setting key cfad73727cbcfbe1420724f9753e9717
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:45.182366 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.169688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:37295 (local address 127.29.3.126:39659)
0811 22:20:45.169827 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.169831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.170898 (+ 1067us) server_negotiation.cc:407] Connection header received
0811 22:20:45.172839 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.172843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.172901 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.172988 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.174398 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.174928 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.176128 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.176362 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.176454 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.178953 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.178956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.178958 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.178986 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.180256 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.180800 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.180803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.180804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.180889 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.181219 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.181222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.181224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.181424 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.181529 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.182063 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.182175 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:45.183336 9252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.170072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:37295)
0811 22:20:45.170700 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.170767 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.172598 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.173158 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.173166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.173628 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.174213 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.174230 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.175078 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.175081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.175990 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.176000 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.176495 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.176498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.176654 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.177250 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.177267 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.178746 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.180384 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.180390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.180404 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.180683 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.180960 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.180962 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.180964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.181113 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.181534 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.181540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.181809 (+ 269us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.181967 (+ 158us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.182258 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":171,"threads_started":1}
I20260811 22:20:45.184804 9249 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39659
I20260811 22:20:45.185086 9249 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:45.185788 9249 heartbeater.cc:511] Master 127.29.3.126:39659 requested a full tablet report, sending...
I20260811 22:20:45.186975 8464 ts_manager.cc:194] Registered new tserver with Master: d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707)
I20260811 22:20:45.187557 8464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:37295
I20260811 22:20:45.197842 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.3.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.125.keytab.
Entry for principal oryx/127.29.3.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.125.keytab.
I20260811 22:20:45.225601 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:44223
--webserver_interface=127.29.3.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.125.keytab
--principal=oryx/127.29.3.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39659,127.29.3.125:44223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:45.341028 9257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:45.341328 9257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:45.341398 9257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:45.345491 9257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:45.345573 9257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:45.345605 9257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:45.345634 9257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:45.345669 9257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:45.351109 9257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39659,127.29.3.125:44223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.125.keytab
--principal=oryx/127.29.3.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.3.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
W20260811 22:20:45.351352 9257 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.3.126:39659,127.29.3.125:44223'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 22:20:45.352314 9257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:45.353359 9257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:45.361927 9270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.361997 9257 server_base.cc:1034] running on GCE node
W20260811 22:20:45.361934 9267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.361934 9268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.362839 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:45.363952 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:45.365113 9257 hybrid_clock.cc:648] HybridClock initialized: now 1786486845365069 us; error 57 us; skew 500 ppm
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:45.367944 9257 init.cc:377] Logged in from keytab as oryx/127.29.3.125@KRBTEST.COM (short username oryx)
I20260811 22:20:45.368901 9257 webserver.cc:460] Webserver started at http://127.29.3.125:34097/ using document root <none> and password file <none>
I20260811 22:20:45.369446 9257 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.369508 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.369692 9257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:45.371847 9257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/instance:
uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "fe54c3f509d8aac0a07d201e01f02283"
server_key_iv: "ebb66439169a36c4f4a0fbd4b866372e"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.372394 9257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal/instance:
uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "fe54c3f509d8aac0a07d201e01f02283"
server_key_iv: "ebb66439169a36c4f4a0fbd4b866372e"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.376067 9257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260811 22:20:45.378481 9284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.379679 9257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:45.379783 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "fe54c3f509d8aac0a07d201e01f02283"
server_key_iv: "ebb66439169a36c4f4a0fbd4b866372e"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.379870 9257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.408341 9257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:45.411615 9257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:45.411801 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:45.418995 9257 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.125:44223
I20260811 22:20:45.419291 9378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.125:44223 every 8 connection(s)
I20260811 22:20:45.420110 9257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
I20260811 22:20:45.422725 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 9257
I20260811 22:20:45.423153 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:45.428138 9382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.125@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:45.437868 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.424266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36892 (local address 127.29.3.125:44223)
0811 22:20:45.424694 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.424710 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.424749 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:20:45.425414 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.425432 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.425779 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.426058 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.427019 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.428036 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.429155 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.429620 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.429796 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.432906 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.432911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.432921 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.432947 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.434675 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.435108 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.435113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.435118 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.435237 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.435499 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.435502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.435503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.436026 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.436231 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.436517 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.436934 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260811 22:20:45.439395 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.430036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:46142)
0811 22:20:45.430534 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.430589 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.430794 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.431106 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.431113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.431593 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.431753 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.431759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.432684 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.432688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.433395 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.433405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.433964 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.433967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.434054 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.434528 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.434539 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.436021 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.437683 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.437692 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.437697 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.437974 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.438339 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.438342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.438344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.438437 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.438844 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.438849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.438969 (+ 120us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.439106 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.439252 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"spinlock_wait_cycles":1920,"thread_start_us":177,"threads_started":1}
I20260811 22:20:45.439510 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.430117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.29.3.126:39659)
0811 22:20:45.430257 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.430261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.430694 (+ 433us) server_negotiation.cc:407] Connection header received
0811 22:20:45.430910 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.430913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.430959 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.431025 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.431868 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.432551 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.433517 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.433839 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.433949 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.436174 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.436178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.436181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.436211 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.437575 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.438124 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.438128 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.438130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.438223 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.438522 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.438524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.438525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.438746 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.438826 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.439228 (+ 402us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.439360 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:20:45.440568 9382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.125@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
I20260811 22:20:45.448055 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.441041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:36908)
0811 22:20:45.441148 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.441162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.441233 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.441402 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.441404 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.441531 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.441626 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.441630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.442730 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.442733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.443247 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.443256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.443647 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.443649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.443723 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.444120 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.444131 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.445318 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.446739 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.446741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.446743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.446919 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.447119 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.447120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.447122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.447188 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.447528 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.447531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.447622 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.447732 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.447874 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260811 22:20:45.448055 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.441106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.29.3.125:44223)
0811 22:20:45.441232 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.441235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.441246 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:20:45.441293 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.441295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.441328 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.441380 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.441706 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.442614 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.443344 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.443564 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.443639 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.445441 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.445444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.445446 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.445468 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.446634 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.446991 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.446993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.446995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.447059 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.447250 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.447252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.447253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.447433 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.447514 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.447794 (+ 280us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.447930 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":62}
I20260811 22:20:45.451152 9382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Bootstrap starting.
I20260811 22:20:45.453557 9382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:45.454373 9382 log.cc:824] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:45.456338 9382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: No bootstrap required, opened a new log
I20260811 22:20:45.459437 9382 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } }
I20260811 22:20:45.459666 9382 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:45.459724 9382 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa74a57e54cf4161bd32e45f09aacdb2, State: Initialized, Role: FOLLOWER
I20260811 22:20:45.460178 9382 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } }
I20260811 22:20:45.460678 9392 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } } }
I20260811 22:20:45.460794 9392 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:45.461436 9382 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:20:45.466192 9421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:20:45.466269 9421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:20:45.468377 9257 master_runner.cc:428] Detected that this master aa74a57e54cf4161bd32e45f09aacdb2 is joining an existing cluster
I20260811 22:20:45.468447 9257 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.3.125:44223
I20260811 22:20:45.478543 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.470910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36916 (local address 127.29.3.125:44223)
0811 22:20:45.471015 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.471019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.471470 (+ 451us) server_negotiation.cc:407] Connection header received
0811 22:20:45.471606 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.471609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.471651 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.471729 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.472377 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.473158 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.474357 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.474596 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.474690 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.475645 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.475648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.475651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.475673 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.477007 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.477377 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.477381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.477382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.477449 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.477699 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.477703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.477705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.477869 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.477952 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.478204 (+ 252us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.478381 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":36}
I20260811 22:20:45.478543 9432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.471092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:36916)
0811 22:20:45.471330 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.471342 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.471524 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.471766 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.471769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.472063 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.472256 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.472264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.473317 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.473320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.474205 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.474226 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.474701 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.474704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.474808 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.475173 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.475187 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.475532 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.477107 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.477110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.477112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.477294 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.477527 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.477530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.477532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.477612 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.477956 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.477959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.478052 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.478137 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.478279 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"spinlock_wait_cycles":6144,"thread_start_us":88,"threads_started":1}
I20260811 22:20:45.479604 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.470963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46152 (local address 127.29.3.126:39659)
0811 22:20:45.471093 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.471097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.471416 (+ 319us) server_negotiation.cc:407] Connection header received
0811 22:20:45.471673 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.471677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.471728 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.471794 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.472377 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.472961 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.473812 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.474026 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.474115 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.475646 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.475651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.475654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.475687 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.477292 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.478097 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.478101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.478103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.478195 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.478632 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.478635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.478637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.478886 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.478961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.479322 (+ 361us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.479449 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:20:45.480123 9431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.470888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:46152)
0811 22:20:45.471271 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.471287 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.471527 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.471897 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.471899 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.472092 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.472256 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.472262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.473071 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.473074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.473689 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.473698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.474149 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.474152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.474250 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.474983 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.474993 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.475449 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.477417 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.477420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.477422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.477982 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.478341 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.478344 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.478346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.478490 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.479034 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.479037 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.479126 (+ 89us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.479802 (+ 676us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.479943 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":42,"spinlock_wait_cycles":3968,"thread_start_us":125,"threads_started":1}
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.126@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
I20260811 22:20:45.493909 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.484556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36922 (local address 127.29.3.125:44223)
0811 22:20:45.484679 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.484684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.485007 (+ 323us) server_negotiation.cc:407] Connection header received
0811 22:20:45.485190 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.485193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.485259 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.485323 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.485969 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.486502 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.487364 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.487575 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.487671 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.490052 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.490056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.490059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.490082 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.491638 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.492232 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.492236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.492238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.492327 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.492694 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.492698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.492700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.493082 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.493182 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.493615 (+ 433us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.493747 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 22:20:45.493956 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.484537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:36922)
0811 22:20:45.484891 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.484933 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.485065 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.485377 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.485383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.485741 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.485851 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.485857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.486602 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.486605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.487246 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.487255 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.487680 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.487683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.487794 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.488286 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.488299 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.489908 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.491768 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.491780 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.491788 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.492122 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.492483 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.492486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.492489 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.492591 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.493194 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.493203 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.493381 (+ 178us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.493544 (+ 163us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.493772 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260811 22:20:45.495818 8464 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.3.125:44223: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
member_type: NON_VOTER
last_known_addr {
host: "127.29.3.125"
port: 44223
}
attrs {
promote: true
}
}
dest_uuid: "4920be7096e146b48b848d87c4371cdb"
cas_config_opid_index: -1
I20260811 22:20:45.496558 8464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } }
I20260811 22:20:45.498852 9439 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER aa74a57e54cf4161bd32e45f09aacdb2 (127.29.3.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } }
I20260811 22:20:45.499567 9332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:45.500530 9439 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.3.125:44223
I20260811 22:20:45.500748 9439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } }
I20260811 22:20:45.500833 9439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:45.501057 9332 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Refusing update from remote peer 4920be7096e146b48b848d87c4371cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
WARNING: no policy specified for oryx/127.29.3.124@KRBTEST.COM; defaulting to no policy
I20260811 22:20:45.501848 9257 master.cc:537] Master@127.29.3.125:44223 shutting down...
I20260811 22:20:45.502411 9440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.124@KRBTEST.COM" created.
W20260811 22:20:45.505257 8441 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260811 22:20:45.506210 8441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb -> Peer aa74a57e54cf4161bd32e45f09aacdb2 (127.29.3.125:44223): Couldn't send request to peer aa74a57e54cf4161bd32e45f09aacdb2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 22:20:45.507439 9257 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:45.507718 9257 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:45.507822 9257 tablet_replica.cc:330] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: stopping tablet replica
I20260811 22:20:45.512362 9257 master.cc:552] Master@127.29.3.125:44223 shutdown complete.
I20260811 22:20:45.512418 9257 master_runner.cc:305] Clearing existing system tablet
I20260811 22:20:45.512651 9257 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.512710 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.514000 9257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260811 22:20:45.514945 9452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.515179 9257 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.515313 9257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:45.515353 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "fe54c3f509d8aac0a07d201e01f02283"
server_key_iv: "ebb66439169a36c4f4a0fbd4b866372e"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.518159 9257 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.124.keytab.
Entry for principal oryx/127.29.3.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.124.keytab.
I20260811 22:20:45.520947 9257 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 22:20:45.521039 9257 log.cc:1196] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 22:20:45.521293 9257 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Deleting consensus metadata
I20260811 22:20:45.521442 9257 master_runner.cc:315] Copying system tablet from 127.29.3.126:39659
I20260811 22:20:45.521576 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:32799
--webserver_interface=127.29.3.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.124.keytab
--principal=oryx/127.29.3.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:20:45.522670 9257 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: tablet copy: Beginning tablet copy session from remote peer at address 127.29.3.126:39659
I20260811 22:20:45.534884 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.523689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.29.3.126:39659)
0811 22:20:45.523818 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.523823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.526613 (+ 2790us) server_negotiation.cc:407] Connection header received
0811 22:20:45.526850 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.526855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.526943 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.527037 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.527597 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.528397 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.529584 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.529928 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.530043 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.531378 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.531383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.531386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.531419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.533132 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.533609 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.533613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.533615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.533709 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.533975 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.533978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.533980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.534192 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.534284 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.534596 (+ 312us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.534724 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:45.534890 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.523593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:46168)
0811 22:20:45.526483 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.526500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.526637 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.527088 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.527092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.527331 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.527469 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.527476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.528515 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.528519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.529451 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.529461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.530051 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.530055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.530156 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.530774 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.530788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.531223 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.533259 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.533263 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.533265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.533514 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.533797 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.533800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.533802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.533892 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.534284 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.534310 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.534424 (+ 114us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.534533 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.534678 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2805,"thread_start_us":91,"threads_started":1}
I20260811 22:20:45.535612 8501 tablet_copy_service.cc:140] P 4920be7096e146b48b848d87c4371cdb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer aa74a57e54cf4161bd32e45f09aacdb2 ({username='oryx', principal='oryx/127.29.3.125@KRBTEST.COM'} at 127.0.0.1:46168)
I20260811 22:20:45.535835 8501 tablet_copy_service.cc:161] P 4920be7096e146b48b848d87c4371cdb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer aa74a57e54cf4161bd32e45f09aacdb2 at {username='oryx', principal='oryx/127.29.3.125@KRBTEST.COM'} at 127.0.0.1:46168: session id = aa74a57e54cf4161bd32e45f09aacdb2-00000000000000000000000000000000
I20260811 22:20:45.537703 8501 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 22:20:45.539525 9257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 22:20:45.541905 9257 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: tablet copy: Starting download of 0 data blocks...
I20260811 22:20:45.542114 9257 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: tablet copy: Starting download of 1 WAL segments...
I20260811 22:20:45.543962 9257 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 22:20:45.545866 8501 tablet_copy_service.cc:342] P 4920be7096e146b48b848d87c4371cdb: Request end of tablet copy session aa74a57e54cf4161bd32e45f09aacdb2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.3.125@KRBTEST.COM'} at 127.0.0.1:46168
I20260811 22:20:45.546003 8501 tablet_copy_service.cc:433] P 4920be7096e146b48b848d87c4371cdb: ending tablet copy session aa74a57e54cf4161bd32e45f09aacdb2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer aa74a57e54cf4161bd32e45f09aacdb2
I20260811 22:20:45.553498 9257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:45.555267 9465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.555513 9257 server_base.cc:1034] running on GCE node
W20260811 22:20:45.563292 9469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.564206 9466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.564465 9257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:45.564842 9257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:45.566012 9257 hybrid_clock.cc:648] HybridClock initialized: now 1786486845565988 us; error 42 us; skew 500 ppm
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:45.567723 9257 init.cc:377] Logged in from keytab as oryx/127.29.3.125@KRBTEST.COM (short username oryx)
I20260811 22:20:45.568262 9257 webserver.cc:460] Webserver started at http://127.29.3.125:41711/ using document root <none> and password file <none>
I20260811 22:20:45.568461 9257 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.568506 9257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.572017 9257 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260811 22:20:45.573173 9482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.573447 9257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:45.573530 9257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "fe54c3f509d8aac0a07d201e01f02283"
server_key_iv: "ebb66439169a36c4f4a0fbd4b866372e"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.573616 9257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.583717 9257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:45.585837 9257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:45.591143 9257 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.125:44223
I20260811 22:20:45.591171 9570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.125:44223 every 8 connection(s)
I20260811 22:20:45.591784 9257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
I20260811 22:20:45.593850 9572 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:20:45.594914 9572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Bootstrap starting.
I20260811 22:20:45.605424 9572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:20:45.605762 9572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Bootstrap complete.
I20260811 22:20:45.606416 9572 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } }
I20260811 22:20:45.606557 9572 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa74a57e54cf4161bd32e45f09aacdb2, State: Initialized, Role: LEARNER
I20260811 22:20:45.606674 9572 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } }
I20260811 22:20:45.607048 9576 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } }
I20260811 22:20:45.607177 9572 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:45.607226 9576 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: LEARNER
I20260811 22:20:45.610675 9607 catalog_manager.cc:1259] Loaded cluster ID: d1e47123fc144161a2424f9e06620014
I20260811 22:20:45.610736 9607 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: loading cluster ID for follower catalog manager: success
I20260811 22:20:45.612816 9607 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: acquiring CA information for follower catalog manager: success
I20260811 22:20:45.614179 9607 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 22:20:45.664184 9461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:45.664443 9461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:45.664507 9461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:45.668540 9461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:45.668603 9461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:45.668637 9461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:45.668663 9461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:45.668697 9461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:45.674108 9461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.124.keytab
--principal=oryx/127.29.3.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:32799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.3.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:45.675213 9461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:45.676206 9461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:45.684931 9615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.684947 9614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.685006 9617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.684940 9461 server_base.cc:1034] running on GCE node
I20260811 22:20:45.685745 9461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:45.686893 9461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:45.688050 9461 hybrid_clock.cc:648] HybridClock initialized: now 1786486845688028 us; error 33 us; skew 500 ppm
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:45.691048 9461 init.cc:377] Logged in from keytab as oryx/127.29.3.124@KRBTEST.COM (short username oryx)
I20260811 22:20:45.692004 9461 webserver.cc:460] Webserver started at http://127.29.3.124:33747/ using document root <none> and password file <none>
I20260811 22:20:45.692617 9461 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.692687 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.692909 9461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:45.695086 9461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/instance:
uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "2ee0c5fe0b0e12c81ba8c162382d0a3b"
server_key_iv: "09f05d94a32d526605230320b285f575"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.695616 9461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal/instance:
uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "2ee0c5fe0b0e12c81ba8c162382d0a3b"
server_key_iv: "09f05d94a32d526605230320b285f575"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.699986 9461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:20:45.702747 9625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.703924 9461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:45.704031 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "2ee0c5fe0b0e12c81ba8c162382d0a3b"
server_key_iv: "09f05d94a32d526605230320b285f575"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.704134 9461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.721524 8783 heartbeater.cc:503] Master 127.29.3.126:39659 was elected leader, sending a full tablet report...
I20260811 22:20:45.722018 9461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:45.725085 9461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:45.725318 9461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:45.735939 9461 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.124:32799
I20260811 22:20:45.736203 9714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.124:32799 every 8 connection(s)
I20260811 22:20:45.737028 9461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
I20260811 22:20:45.738857 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 9461
I20260811 22:20:45.740285 9715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:45.745080 9715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.3.124@KRBTEST.COM
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.124@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:45.754485 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.740354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.29.3.124:32799)
0811 22:20:45.740832 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.740873 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.740911 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:20:45.742160 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.742188 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.742597 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.742888 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.743855 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.745021 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.746174 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.746776 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.746929 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.749698 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.749704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.749728 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.749761 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.751649 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.752065 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.752070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.752074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.752183 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.752454 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.752458 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.752459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.752861 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.753062 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.753276 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.753538 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":150,"threads_started":1}
I20260811 22:20:45.756343 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.747261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46182 (local address 127.29.3.126:39659)
0811 22:20:45.747419 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.747423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.747932 (+ 509us) server_negotiation.cc:407] Connection header received
0811 22:20:45.748056 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.748059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.748109 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.748188 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.749042 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.749625 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.750876 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.751206 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.751332 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.753310 (+ 1978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.753313 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.753316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.753347 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.754723 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.755141 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.755144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.755146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.755240 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.755493 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.755496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.755498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.755673 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.755761 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.756078 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.756197 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:20:45.756402 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.747192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:46182)
0811 22:20:45.747807 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.747846 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.747964 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.748240 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.748246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.748622 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.748897 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.748907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.749803 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.749805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.750745 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.750757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.751324 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.751327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.751410 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.751836 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.751848 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.753159 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.754823 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.754831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.754837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.755043 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.755327 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.755330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.755332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.755410 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.755775 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.755782 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.755886 (+ 104us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.756004 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.756180 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":332,"threads_started":1}
I20260811 22:20:45.757313 9715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.124@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
I20260811 22:20:45.764227 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.757756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:36928)
0811 22:20:45.757876 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.757890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.757962 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.758357 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.758360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.758509 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.758613 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.758620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.759439 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.759441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.759942 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.759948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.760311 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.760314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.760377 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.760709 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.760721 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.761774 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.763173 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.763176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.763177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.763350 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.763554 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.763556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.763557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.763619 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.763897 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.763899 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.763966 (+ 67us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.764036 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.764123 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 22:20:45.764299 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.757887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.29.3.125:44223)
0811 22:20:45.758141 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.758144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.758155 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:20:45.758197 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.758199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.758237 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.758346 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.758713 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.759364 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.760018 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.760237 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.760304 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.761898 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.761900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.761903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.761925 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.763084 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.763423 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.763425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.763426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.763493 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.763681 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.763683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.763684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.763834 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.763905 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.764078 (+ 173us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.764174 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":108,"threads_started":1}
I20260811 22:20:45.764932 9715 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.124@KRBTEST.COM for oryx/127.29.3.124@KRBTEST.COM
I20260811 22:20:45.772064 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.765333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:35440)
0811 22:20:45.765427 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.765439 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.765503 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.765684 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.765686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.765805 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.765872 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.765876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.767205 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.767208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.767675 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.767682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.768034 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.768037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.768129 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.768489 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.768500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.769531 (+ 1031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.770975 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.770977 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.770978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.771151 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.771397 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.771399 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.771400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.771468 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.771716 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.771718 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.771784 (+ 66us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.771850 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.771934 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":24}
I20260811 22:20:45.772102 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.765393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35440 (local address 127.29.3.124:32799)
0811 22:20:45.765489 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.765492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.765504 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:45.765574 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.765576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.765606 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.765663 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.765948 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.767107 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.767763 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.767964 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.768027 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.769641 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.769644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.769645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.769666 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.770886 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.771254 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.771257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.771258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.771328 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.771520 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.771522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.771523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.771658 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.771731 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.771891 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.771980 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":39}
I20260811 22:20:45.774927 9715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Bootstrap starting.
I20260811 22:20:45.777208 9715 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:45.777884 9715 log.cc:824] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:45.779814 9715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: No bootstrap required, opened a new log
I20260811 22:20:45.783270 9715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } }
I20260811 22:20:45.783464 9715 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:45.783504 9715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0db3c0a21e4b48cd9a63a1723f6eb85a, State: Initialized, Role: FOLLOWER
I20260811 22:20:45.783864 9715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } }
I20260811 22:20:45.784303 9728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } } }
I20260811 22:20:45.784456 9728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:45.784956 9715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:20:45.789503 9744 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:20:45.789587 9744 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:20:45.792754 9461 master_runner.cc:428] Detected that this master 0db3c0a21e4b48cd9a63a1723f6eb85a is joining an existing cluster
I20260811 22:20:45.792804 9461 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.3.124:32799
I20260811 22:20:45.803198 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.795146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:36934)
0811 22:20:45.795454 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.795466 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.795579 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.795891 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.795893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.796154 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.796379 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.796388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.797434 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.797437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.798274 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.798282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.798739 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.798743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.798834 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.799767 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.799779 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.800114 (+ 335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.801680 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.801683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.801685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.801882 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.802135 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.802139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.802141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.802236 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.802607 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.802611 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.802707 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.802855 (+ 148us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.802998 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":37,"spinlock_wait_cycles":17280,"thread_start_us":106,"threads_started":1}
I20260811 22:20:45.803520 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.795005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.29.3.125:44223)
0811 22:20:45.795113 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.795117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.795527 (+ 410us) server_negotiation.cc:407] Connection header received
0811 22:20:45.795668 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.795670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.795726 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.795771 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.796525 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.797312 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.798398 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.798615 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.798701 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.800256 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.800259 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.800261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.800283 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.801574 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.801974 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.801977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.801978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.802050 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.802337 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.802341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.802342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.802513 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.802601 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.802945 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.803088 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 22:20:45.808014 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 8567
I20260811 22:20:45.823494 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.794936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46196 (local address 127.29.3.126:39659)
0811 22:20:45.795059 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.795063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.795426 (+ 363us) server_negotiation.cc:407] Connection header received
0811 22:20:45.795606 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.795610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.795660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.795708 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.796521 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.797057 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.798250 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.798614 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.798743 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.820531 (+ 21788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.820535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.820538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.820572 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.821896 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.822351 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.822354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.822356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.822430 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.822669 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.822671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.822673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.822877 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.822934 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.823236 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.823348 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:45.823993 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.794893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:46196)
0811 22:20:45.795339 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.795355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.795498 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.795858 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.795861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.796151 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.796379 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.796388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.797169 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.797172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.798108 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.798119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.798873 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.798877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.798976 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.819885 (+ 20909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.819900 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.820323 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.822031 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.822034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.822036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.822238 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.822517 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.822519 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.822521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.822587 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.822994 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.822996 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.823080 (+ 84us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.823700 (+ 620us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.823835 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":30,"thread_start_us":164,"threads_started":1}
I20260811 22:20:45.845328 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.795517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.29.3.124:32799)
0811 22:20:45.795612 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.795616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.802720 (+ 7104us) server_negotiation.cc:407] Connection header received
0811 22:20:45.802818 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.802822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.802865 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.802925 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.803446 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.803998 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.804862 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.805070 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.805150 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.806114 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.806117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.806120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.806143 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.807601 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.808013 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.808016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.808018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.808098 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.844374 (+ 36276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.844377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.844379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.844623 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.844695 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.845008 (+ 313us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.845132 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:20:45.845863 9754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.795703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:35452)
0811 22:20:45.802582 (+ 6879us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.802597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.802745 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.803013 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.803015 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.803205 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.803343 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.803354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.804102 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.804103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.804752 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.804763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.805186 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.805189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.805286 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.805670 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.805682 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.805990 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.807693 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.807696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.807698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.807924 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.844101 (+ 36177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.844105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.844107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.844270 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.844754 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.844756 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.844845 (+ 89us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.845546 (+ 701us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.845692 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6808,"thread_start_us":81,"threads_started":1}
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486844, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.126@KRBTEST.COM for oryx/127.29.3.124@KRBTEST.COM
I20260811 22:20:45.851291 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:33273
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:39659
--webserver_port=38475
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:20:45.853329 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.827626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:35462)
0811 22:20:45.842308 (+ 14682us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.842327 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.842458 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.844548 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.844551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.844793 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.844919 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.844928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.846946 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.846949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.847529 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.847540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.847997 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.848000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.848098 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.848576 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.848588 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.849794 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.851729 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.851733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.851736 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.851995 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.852303 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.852306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.852308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.852395 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.852786 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.852789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.852898 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.853005 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.853151 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14534}
I20260811 22:20:45.853377 9757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.830619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35462 (local address 127.29.3.124:32799)
0811 22:20:45.844213 (+ 13594us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.844222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.844235 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:45.844334 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.844337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.844397 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.845029 (+ 632us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.845135 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.846828 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.847656 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.847880 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.847961 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.849928 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.849932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.849934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.849955 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.851569 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.852101 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.852105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.852107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.852207 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.852480 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.852483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.852485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.852682 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.852789 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.853073 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.853192 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13518,"thread_start_us":684,"threads_started":1}
I20260811 22:20:45.854879 8464 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.3.124:32799: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
member_type: NON_VOTER
last_known_addr {
host: "127.29.3.124"
port: 32799
}
attrs {
promote: true
}
}
dest_uuid: "4920be7096e146b48b848d87c4371cdb"
cas_config_opid_index: 5
I20260811 22:20:45.855536 8464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } }
I20260811 22:20:45.856747 9440 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.858220 9440 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.3.124:32799
I20260811 22:20:45.858451 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.858558 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:45.859300 9461 master.cc:537] Master@127.29.3.124:32799 shutting down...
I20260811 22:20:45.859648 9667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:45.861217 9667 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Refusing update from remote peer 4920be7096e146b48b848d87c4371cdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 22:20:45.862562 9439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 22:20:45.864449 8441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb -> Peer 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124:32799): Couldn't send request to peer 0db3c0a21e4b48cd9a63a1723f6eb85a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 22:20:45.865068 9461 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:45.865387 9461 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:45.865491 9461 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: stopping tablet replica
I20260811 22:20:45.869907 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.861787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36944 (local address 127.29.3.125:44223)
0811 22:20:45.861918 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.861922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.864777 (+ 2855us) server_negotiation.cc:407] Connection header received
0811 22:20:45.865073 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.865076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.865144 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.865249 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:45.865665 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.866500 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.868267 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.868930 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.869008 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.869175 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.869713 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.869784 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:20:45.870604 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.858968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:36944)
0811 22:20:45.864396 (+ 5428us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.864413 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.864929 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.865420 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.865423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.865449 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:45.865542 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.865548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.866643 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.866646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.868120 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.868132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.869274 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.869278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.869375 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.869422 (+ 47us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.870225 (+ 803us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.870479 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5337}
I20260811 22:20:45.871169 9529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 LEARNER]: Refusing update from remote peer 4920be7096e146b48b848d87c4371cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 22:20:45.871615 9439 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 22:20:45.873657 9529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.876047 9576 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4920be7096e146b48b848d87c4371cdb. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.876170 9576 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: LEARNER
I20260811 22:20:45.876574 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: NON_VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: true } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.876745 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: LEARNER
I20260811 22:20:45.876794 9439 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: attempting to promote NON_VOTER aa74a57e54cf4161bd32e45f09aacdb2 to VOTER
I20260811 22:20:45.877264 9439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } }
I20260811 22:20:45.878671 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.878309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:35470)
0811 22:20:45.878518 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.878594 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:32799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260811 22:20:45.879053 9529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 LEARNER]: Refusing update from remote peer 4920be7096e146b48b848d87c4371cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 22:20:45.879536 9440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 22:20:45.880220 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.879711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:35474)
0811 22:20:45.880103 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.880154 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:32799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324}
W20260811 22:20:45.880645 8441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb -> Peer 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124:32799): Couldn't send request to peer 0db3c0a21e4b48cd9a63a1723f6eb85a. Status: Network error: Client connection negotiation failed: client connection to 127.29.3.124:32799: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 22:20:45.881556 9461 master.cc:552] Master@127.29.3.124:32799 shutdown complete.
I20260811 22:20:45.881608 9461 master_runner.cc:305] Clearing existing system tablet
I20260811 22:20:45.881839 9461 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.881894 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.883139 9440 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, aa74a57e54cf4161bd32e45f09aacdb2 (127.29.3.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.884649 9440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.884748 9440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:45.884960 9461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:45.885826 9439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.885923 9439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:45.886524 9775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.886704 9461 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.886806 9461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:45.886835 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "2ee0c5fe0b0e12c81ba8c162382d0a3b"
server_key_iv: "09f05d94a32d526605230320b285f575"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.886893 9529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, aa74a57e54cf4161bd32e45f09aacdb2 (127.29.3.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.888957 9461 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:45.888989 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:45.889091 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:45.891325 9461 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 22:20:45.891429 9461 log.cc:1196] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 22:20:45.891729 9461 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Deleting consensus metadata
I20260811 22:20:45.891882 9461 master_runner.cc:315] Copying system tablet from 127.29.3.126:39659
I20260811 22:20:45.893046 9461 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: tablet copy: Beginning tablet copy session from remote peer at address 127.29.3.126:39659
I20260811 22:20:45.911254 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.896398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.29.3.126:39659)
0811 22:20:45.896556 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:45.896561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:45.901099 (+ 4538us) server_negotiation.cc:407] Connection header received
0811 22:20:45.901247 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:45.901251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:45.901309 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:45.901386 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:45.902051 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.902893 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.904076 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.904392 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.904466 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.905920 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:45.905924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:45.905927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:45.905955 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:45.909403 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.909891 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.909895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.909896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.909982 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:45.910272 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:45.910275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:45.910277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:45.910530 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:45.910601 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:45.910967 (+ 366us) server_negotiation.cc:299] Negotiation successful
0811 22:20:45.911088 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:20:45.912073 9784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:45.893745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:46202)
0811 22:20:45.900983 (+ 7238us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:45.900999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:45.901132 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:45.901521 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:45.901524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:45.901765 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:45.901907 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.901915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.903042 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.903045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.903938 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:45.903949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:45.904648 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:45.904652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:45.904744 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:45.905312 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:45.905327 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:45.905755 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:45.909523 (+ 3768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.909526 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.909528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.909781 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.910086 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:45.910090 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:45.910092 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:45.910175 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:45.910685 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:45.910688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:45.910788 (+ 100us) client_negotiation.cc:769] Sending connection context
0811 22:20:45.911719 (+ 931us) client_negotiation.cc:240] Negotiation successful
0811 22:20:45.911878 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7155,"thread_start_us":2413,"threads_started":1}
I20260811 22:20:45.913326 8501 tablet_copy_service.cc:140] P 4920be7096e146b48b848d87c4371cdb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0db3c0a21e4b48cd9a63a1723f6eb85a ({username='oryx', principal='oryx/127.29.3.124@KRBTEST.COM'} at 127.0.0.1:46202)
I20260811 22:20:45.913424 8501 tablet_copy_service.cc:161] P 4920be7096e146b48b848d87c4371cdb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0db3c0a21e4b48cd9a63a1723f6eb85a at {username='oryx', principal='oryx/127.29.3.124@KRBTEST.COM'} at 127.0.0.1:46202: session id = 0db3c0a21e4b48cd9a63a1723f6eb85a-00000000000000000000000000000000
I20260811 22:20:45.914649 8501 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 22:20:45.916064 9461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 22:20:45.919186 9461 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: tablet copy: Starting download of 0 data blocks...
I20260811 22:20:45.919390 9461 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: tablet copy: Starting download of 1 WAL segments...
I20260811 22:20:45.920929 9461 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 22:20:45.922709 8501 tablet_copy_service.cc:342] P 4920be7096e146b48b848d87c4371cdb: Request end of tablet copy session 0db3c0a21e4b48cd9a63a1723f6eb85a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.3.124@KRBTEST.COM'} at 127.0.0.1:46202
I20260811 22:20:45.922796 8501 tablet_copy_service.cc:433] P 4920be7096e146b48b848d87c4371cdb: ending tablet copy session 0db3c0a21e4b48cd9a63a1723f6eb85a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0db3c0a21e4b48cd9a63a1723f6eb85a
I20260811 22:20:45.930727 9461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:45.934202 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.947715 9791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:45.948021 9788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:45.950523 9461 server_base.cc:1034] running on GCE node
I20260811 22:20:45.950804 9461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:45.951217 9461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:45.952430 9461 hybrid_clock.cc:648] HybridClock initialized: now 1786486845952418 us; error 33 us; skew 500 ppm
Aug 11 22:20:45 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486845, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:45.953853 9016 heartbeater.cc:503] Master 127.29.3.126:39659 was elected leader, sending a full tablet report...
I20260811 22:20:45.954196 9461 init.cc:377] Logged in from keytab as oryx/127.29.3.124@KRBTEST.COM (short username oryx)
I20260811 22:20:45.955401 9461 webserver.cc:460] Webserver started at http://127.29.3.124:45375/ using document root <none> and password file <none>
I20260811 22:20:45.955686 9461 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:45.955775 9461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:45.957247 9461 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:45.958329 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:45.958566 9461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:45.958652 9461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "2ee0c5fe0b0e12c81ba8c162382d0a3b"
server_key_iv: "09f05d94a32d526605230320b285f575"
server_key_version: "encryptionkey@0"
I20260811 22:20:45.958734 9461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:45.976078 9461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:45.978149 9461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:45.983072 9461 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.124:32799
I20260811 22:20:45.983108 9882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.124:32799 every 8 connection(s)
I20260811 22:20:45.983642 9461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
I20260811 22:20:45.985430 9883 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:20:45.986547 9883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Bootstrap starting.
I20260811 22:20:45.998618 9883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:20:45.998924 9883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: Bootstrap complete.
I20260811 22:20:45.999485 9883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } }
I20260811 22:20:45.999648 9883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0db3c0a21e4b48cd9a63a1723f6eb85a, State: Initialized, Role: LEARNER
I20260811 22:20:45.999785 9883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } }
I20260811 22:20:46.000156 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:46.000301 9883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:46.000377 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: This master's current role is: LEARNER
I20260811 22:20:46.004495 9908 catalog_manager.cc:1259] Loaded cluster ID: d1e47123fc144161a2424f9e06620014
I20260811 22:20:46.004555 9908 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: loading cluster ID for follower catalog manager: success
I20260811 22:20:46.006836 9908 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: acquiring CA information for follower catalog manager: success
I20260811 22:20:46.008355 9908 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 22:20:46.017697 9761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:46.017935 9761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:46.017982 9761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:46.022161 9761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:46.022217 9761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:46.022281 9761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:46.027707 9761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:33273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=38475
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:46.028872 9761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:46.029918 9761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:46.040059 9915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:46.040123 9914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:46.040100 9917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:46.040505 9761 server_base.cc:1034] running on GCE node
I20260811 22:20:46.040938 9761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:46.041577 9761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:46.042748 9761 hybrid_clock.cc:648] HybridClock initialized: now 1786486846042724 us; error 37 us; skew 500 ppm
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:46.045429 9761 init.cc:377] Logged in from keytab as oryx/127.29.3.65@KRBTEST.COM (short username oryx)
I20260811 22:20:46.046458 9761 webserver.cc:460] Webserver started at http://127.29.3.65:38475/ using document root <none> and password file <none>
I20260811 22:20:46.047024 9761 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:46.047125 9761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:46.051335 9761 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:46.053356 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.054661 9761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:46.054768 9761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "aed735695b25402aa239b0452e6ce7e6"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "5a4ae9a394f149900246105b85ab85e2"
server_key_iv: "0c1ac9fabbea0d6c43f918ea3b7d81cf"
server_key_version: "encryptionkey@0"
I20260811 22:20:46.055294 9761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:46.082370 9761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:46.085151 9761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:46.085343 9761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:46.085948 9761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:46.086887 9761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:46.086936 9761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.086992 9761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:46.087021 9761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.097671 9761 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:33273
I20260811 22:20:46.097694 10134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:33273 every 8 connection(s)
I20260811 22:20:46.098742 9761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:46.100293 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 9761
I20260811 22:20:46.100492 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 8791
I20260811 22:20:46.112025 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:46393
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:39659
--webserver_port=37329
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.124@KRBTEST.COM
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:46.119560 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.105165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:59375 (local address 127.29.3.126:39659)
0811 22:20:46.105338 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.105343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.105360 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:20:46.105577 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.105580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.105634 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.105716 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.107140 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.107979 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.111191 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.111969 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.112056 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.115236 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.115240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.115242 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.115271 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.117018 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.117670 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.117674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.117676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.117765 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118199 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.118203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.118204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.118479 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.118566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.119256 (+ 690us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.119390 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 22:20:46.119716 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.104473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:50445 (local address 127.29.3.125:44223)
0811 22:20:46.104612 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.104616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.108057 (+ 3441us) server_negotiation.cc:407] Connection header received
0811 22:20:46.108224 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.108227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.108282 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.108330 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.108907 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.109682 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.110740 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.111071 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.111160 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.115574 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.115578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.115581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.115611 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.117783 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118323 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.118327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.118328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.118421 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118749 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.118752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.118753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.118976 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.119058 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.119428 (+ 370us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.119555 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:20:46.120508 10143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.104644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:59375)
0811 22:20:46.104956 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.104970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.105401 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.105859 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.105866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.106228 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.106955 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.106976 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.108952 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.108956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.111065 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.111074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.112227 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.112231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.112320 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.112807 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.112822 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.115037 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.117159 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.117166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.117184 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.117532 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.117893 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.117895 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.117897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.118089 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.118622 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.118629 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.118983 (+ 354us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.119165 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.119523 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":7,"thread_start_us":127,"threads_started":1}
I20260811 22:20:46.120786 10145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.105157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.29.3.65:50445)
0811 22:20:46.107972 (+ 2815us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.107985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.108123 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.108458 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.108461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.108637 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.108769 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.108775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.109809 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.109813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.110617 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.110627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.111332 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.111335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.111435 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.111895 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.111911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.115038 (+ 3127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.117914 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.117916 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.117919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.118175 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.118540 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.118544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.118636 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.119133 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.119136 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.119234 (+ 98us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.119881 (+ 647us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.120096 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2756,"thread_start_us":128,"threads_started":1}
I20260811 22:20:46.121894 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.101150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.29.3.65:49345)
0811 22:20:46.103919 (+ 2769us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.103981 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.105332 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.107195 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.107198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.107531 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.107729 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.107736 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.108987 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.108990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.110211 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.110223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.110925 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.110929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.111162 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.111828 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.111851 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.115037 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.117651 (+ 2614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.117654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.117656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.117896 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.118193 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.118197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.118281 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.118775 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.118777 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.121503 (+ 2726us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.121606 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.121741 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2445,"mutex_wait_us":2588,"spinlock_wait_cycles":11648,"thread_start_us":2174,"threads_started":1}
I20260811 22:20:46.122021 10139 heartbeater.cc:347] Connected to a master server at 127.29.3.125:44223
I20260811 22:20:46.122017 10138 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39659
I20260811 22:20:46.122411 10139 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.122705 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.105056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:49345 (local address 127.29.3.124:32799)
0811 22:20:46.106876 (+ 1820us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.106881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.106896 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:46.106938 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.106941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.106988 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.107070 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.107862 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.108850 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.110374 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.110735 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.111484 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.115952 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.115956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.115959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.116016 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.117524 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118011 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.118014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.118016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.118096 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.118436 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.118439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.118441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.118657 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.118919 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.122402 (+ 3483us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.122544 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1741,"thread_start_us":82,"threads_started":1}
I20260811 22:20:46.123011 10139 heartbeater.cc:511] Master 127.29.3.125:44223 requested a full tablet report, sending...
I20260811 22:20:46.122411 10138 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.123600 10138 heartbeater.cc:511] Master 127.29.3.126:39659 requested a full tablet report, sending...
I20260811 22:20:46.124473 10137 heartbeater.cc:347] Connected to a master server at 127.29.3.124:32799
I20260811 22:20:46.124558 10137 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.124662 8464 ts_manager.cc:194] Re-registered known tserver with Master: aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:46.124744 10137 heartbeater.cc:511] Master 127.29.3.124:32799 requested a full tablet report, sending...
I20260811 22:20:46.124804 9515 ts_manager.cc:194] Registered new tserver with Master: aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:46.125440 8464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:59375
I20260811 22:20:46.125790 9822 ts_manager.cc:194] Registered new tserver with Master: aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:46.189745 9249 heartbeater.cc:503] Master 127.29.3.126:39659 was elected leader, sending a full tablet report...
W20260811 22:20:46.228780 10146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:46.229068 10146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:46.229132 10146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:46.233393 10146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:46.233462 10146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:46.233542 10146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:46.239024 10146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:46393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=37329
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:46.240075 10146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:46.241062 10146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:46.250602 10155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:46.250609 10156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:46.250666 10162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:46.250805 10146 server_base.cc:1034] running on GCE node
I20260811 22:20:46.251286 10146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:46.251959 10146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:46.253118 10146 hybrid_clock.cc:648] HybridClock initialized: now 1786486846253078 us; error 49 us; skew 500 ppm
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:46.255903 10146 init.cc:377] Logged in from keytab as oryx/127.29.3.66@KRBTEST.COM (short username oryx)
I20260811 22:20:46.256759 10146 webserver.cc:460] Webserver started at http://127.29.3.66:37329/ using document root <none> and password file <none>
I20260811 22:20:46.257334 10146 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:46.257440 10146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:46.261533 10146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:46.264030 10172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.265523 10146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:20:46.266197 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.262218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:35480)
0811 22:20:46.262446 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.262461 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.262549 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.262847 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.262849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.262863 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:46.262954 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.262958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.263912 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.263916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.264918 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.264931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.265871 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.265874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.265948 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.265971 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.266023 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.266088 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260811 22:20:46.266245 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.262375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.29.3.124:32799)
0811 22:20:46.262525 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.262529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.262547 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:20:46.262654 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.262658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.262733 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.262835 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:46.263061 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.263788 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.265051 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.265751 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.265860 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.266015 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.266052 (+ 37us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.266124 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:20:46.265645 10146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "345fc1b58590433a8cb2eca25e4deec7"
format_stamp: "Formatted at 2026-08-11 22:20:44 on dist-test-slave-4tdj"
server_key: "d20fff3c28d60675678bb9dab1e60fa7"
server_key_iv: "84235fc499711116f41bbf2582df2a0b"
server_key_version: "encryptionkey@0"
I20260811 22:20:46.266368 10146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:46.267035 9841 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 22:20:46.267472 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4920be7096e146b48b848d87c4371cdb. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:46.267607 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: This master's current role is: LEARNER
I20260811 22:20:46.307169 10146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:46.310451 10146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:46.310657 10146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:46.311321 10146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:46.312289 10146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:46.312346 10146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.312407 10146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:46.312438 10146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.322229 10146 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:46393
I20260811 22:20:46.322259 10376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:46393 every 8 connection(s)
I20260811 22:20:46.323360 10146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:46.330075 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 10146
I20260811 22:20:46.330256 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 9024
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.124@KRBTEST.COM
I20260811 22:20:46.340059 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.325320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:51539 (local address 127.29.3.126:39659)
0811 22:20:46.325493 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.325499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.326465 (+ 966us) server_negotiation.cc:407] Connection header received
0811 22:20:46.328027 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.328031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.328086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.328172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.329501 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.330186 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.331841 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.332112 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.332173 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.335164 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.335168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.335171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.335199 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.336729 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.337488 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.337492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.337494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.337601 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.338442 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.338446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.338448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.338722 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.338818 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.339696 (+ 878us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.339855 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:20:46.342375 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.325782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:51539)
0811 22:20:46.326268 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.326350 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.327403 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.328207 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.328210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.328594 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.329328 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.329347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.330437 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.330440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.331712 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.331723 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.332369 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.332372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.332583 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.333285 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.333304 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.334970 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.336874 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.336882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.336904 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.337355 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.337725 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.337728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.337730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.338273 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.338897 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.338906 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.339342 (+ 436us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.340263 (+ 921us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.340941 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":103,"spinlock_wait_cycles":90624,"thread_start_us":175,"threads_started":1}
I20260811 22:20:46.344920 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.325381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:43149 (local address 127.29.3.125:44223)
0811 22:20:46.325523 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.325527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.326467 (+ 940us) server_negotiation.cc:407] Connection header received
0811 22:20:46.327638 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.327641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.327701 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.327825 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.329505 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.330337 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.335157 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.335885 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.335979 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.341878 (+ 5899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.341881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.341884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.341911 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.343226 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.343671 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.343674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.343676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.343773 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.344138 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.344140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.344142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.344339 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.344403 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.344684 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.344791 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:20:46.345258 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.327023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:44661 (local address 127.29.3.124:32799)
0811 22:20:46.327147 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.327150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.327164 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:20:46.327755 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.327758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.327809 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.327881 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.329487 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.330337 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.333776 (+ 3439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.334155 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.334246 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.339888 (+ 5642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.339892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.339894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.339922 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.341434 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.343659 (+ 2225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.343663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.343665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.343758 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.344141 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.344145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.344147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.344402 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.344483 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.344900 (+ 417us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.345065 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:20:46.345386 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.325726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.29.3.66:43149)
0811 22:20:46.326268 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.326351 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.327407 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.328032 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.328042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.328632 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.329331 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.329347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.330472 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.330476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.332044 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.332059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.336287 (+ 4228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.336294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.336390 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.337252 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.337264 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.339739 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.343348 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.343350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.343352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.343540 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.343888 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.343891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.343893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.344025 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.344450 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.344452 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.344543 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.345045 (+ 502us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.345174 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":80,"spinlock_wait_cycles":11136,"thread_start_us":225,"threads_started":1}
I20260811 22:20:46.346516 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.326857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.29.3.66:44661)
0811 22:20:46.327089 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.327102 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.327418 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.328038 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.328044 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.328586 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.329326 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.329346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.330472 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.330476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.333636 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.333648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.334443 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.334447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.334647 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.336768 (+ 2121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.336780 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.339401 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.341560 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.341562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.341564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.341754 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.343882 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.343885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.343888 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.344006 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.344568 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.344571 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.344678 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.345508 (+ 830us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.345952 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1767,"spinlock_wait_cycles":37504,"thread_start_us":93,"threads_started":1}
I20260811 22:20:46.347105 10382 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39659
I20260811 22:20:46.347128 10380 heartbeater.cc:347] Connected to a master server at 127.29.3.124:32799
I20260811 22:20:46.347368 10382 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.347368 10380 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.347905 10380 heartbeater.cc:511] Master 127.29.3.124:32799 requested a full tablet report, sending...
I20260811 22:20:46.347904 10382 heartbeater.cc:511] Master 127.29.3.126:39659 requested a full tablet report, sending...
I20260811 22:20:46.349052 8464 ts_manager.cc:194] Re-registered known tserver with Master: 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66:46393)
I20260811 22:20:46.349099 9822 ts_manager.cc:194] Registered new tserver with Master: 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66:46393)
I20260811 22:20:46.349807 8464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:51539
I20260811 22:20:46.350334 10383 heartbeater.cc:347] Connected to a master server at 127.29.3.125:44223
I20260811 22:20:46.350414 10383 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.350584 10383 heartbeater.cc:511] Master 127.29.3.125:44223 requested a full tablet report, sending...
I20260811 22:20:46.351327 9515 ts_manager.cc:194] Registered new tserver with Master: 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66:46393)
I20260811 22:20:46.353173 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:40707
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:39659
--webserver_port=39385
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:46.467012 10393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:46.467273 10393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:46.467332 10393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:46.471403 10393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:46.471463 10393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:46.471544 10393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:46.476769 10393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:40707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=39385
--tserver_master_addrs=127.29.3.126:39659,127.29.3.125:44223,127.29.3.124:32799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:46.477774 10393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:46.478826 10393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:46.488395 10407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:46.488476 10393 server_base.cc:1034] running on GCE node
W20260811 22:20:46.488468 10408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:46.488416 10414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:46.489138 10393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:46.489765 10393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:46.490922 10393 hybrid_clock.cc:648] HybridClock initialized: now 1786486846490899 us; error 41 us; skew 500 ppm
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:46.493888 10393 init.cc:377] Logged in from keytab as oryx/127.29.3.67@KRBTEST.COM (short username oryx)
I20260811 22:20:46.494776 10393 webserver.cc:460] Webserver started at http://127.29.3.67:39385/ using document root <none> and password file <none>
I20260811 22:20:46.495316 10393 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:46.495401 10393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:46.499179 10393 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 22:20:46.501276 10424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.502550 10393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:46.502671 10393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "d8dbab3c846c4825b4f0315139d4c099"
format_stamp: "Formatted at 2026-08-11 22:20:45 on dist-test-slave-4tdj"
server_key: "e58759585696d1cb682d0ed35f14bd3d"
server_key_iv: "76e68b64f5bdea3d851be90257a68b18"
server_key_version: "encryptionkey@0"
I20260811 22:20:46.503188 10393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:46.516794 10393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:46.519654 10393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:46.519829 10393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:46.520413 10393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:46.521268 10393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:46.521317 10393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.521368 10393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:46.521399 10393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:46.529932 10393 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:40707
I20260811 22:20:46.529989 10627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:40707 every 8 connection(s)
I20260811 22:20:46.530974 10393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:46.539986 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 10393
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.124@KRBTEST.COM
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.125@KRBTEST.COM
Aug 11 22:20:46 dist-test-slave-4tdj krb5kdc[8389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486846, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:46.545776 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.534219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:41849 (local address 127.29.3.124:32799)
0811 22:20:46.534358 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.534362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.534376 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:20:46.535053 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.535056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.535126 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.535197 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.536422 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.537260 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538574 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.538876 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538968 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.541573 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.541577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.541580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.541606 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.543224 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.543788 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.543792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.543794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.543883 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.544305 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.544309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.544310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.544533 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.544625 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.545495 (+ 870us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.545614 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 22:20:46.546669 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.532687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:53339 (local address 127.29.3.125:44223)
0811 22:20:46.532834 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.532839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.533688 (+ 849us) server_negotiation.cc:407] Connection header received
0811 22:20:46.534772 (+ 1084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.534776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.534835 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.534912 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.536448 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.537245 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538401 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.538708 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538816 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.541803 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.541807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.541810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.541839 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.543482 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.543944 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.543947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.543949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.544047 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.544350 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.544354 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.544356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.544569 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.544658 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.546403 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.546519 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:20:46.546890 10639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.533107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:39877)
0811 22:20:46.533538 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.533591 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.534576 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.535080 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.535090 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.535663 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.536287 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.536300 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.537180 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.537183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.538439 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.538446 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.539062 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.539066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.539162 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.541746 (+ 2584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.541760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.543106 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.545047 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.545050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.545052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.545296 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.545597 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.545600 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.545602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.545687 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.546106 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.546109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.546188 (+ 79us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.546273 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.546423 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":129,"spinlock_wait_cycles":86272,"thread_start_us":157,"threads_started":1}
I20260811 22:20:46.547427 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.534038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.29.3.67:41849)
0811 22:20:46.534263 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.534276 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.534577 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.535237 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.535239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.535661 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.536281 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.536298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.537521 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.537524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.538426 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.538437 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.538970 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.539070 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.539625 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.539646 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.541379 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.543336 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.543342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.543357 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.543674 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.543991 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.543994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.543996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.544176 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.544682 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.544689 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.545022 (+ 333us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.545251 (+ 229us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.547019 (+ 1768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1565,"spinlock_wait_cycles":9984,"thread_start_us":82,"threads_started":1}
I20260811 22:20:46.547569 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.532664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:39877 (local address 127.29.3.126:39659)
0811 22:20:46.532819 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.532824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.533692 (+ 868us) server_negotiation.cc:407] Connection header received
0811 22:20:46.534874 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.534877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.534929 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.535013 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.536442 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.537019 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538574 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.538828 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538930 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.543267 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.543271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.543274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.543303 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.544920 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.545401 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.545405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.545406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.545495 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.545804 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.545807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.545809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.546005 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.546092 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.546388 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.546495 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:20:46.547655 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.533068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.29.3.67:53339)
0811 22:20:46.533538 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.533591 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.534569 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.535081 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.535089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.535665 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:46.536290 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.536299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.537358 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.537362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.538269 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.538279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.538855 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.538859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.539042 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.539838 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:46.539853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:46.541652 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:46.543605 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.543608 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.543610 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.543845 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.544154 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:46.544157 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:46.544159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:46.544247 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:46.544707 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:46.544710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:46.545025 (+ 315us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.545249 (+ 224us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.545568 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":202,"spinlock_wait_cycles":12672,"thread_start_us":122,"threads_started":1}
I20260811 22:20:46.548517 10633 heartbeater.cc:347] Connected to a master server at 127.29.3.125:44223
I20260811 22:20:46.548553 10630 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39659
I20260811 22:20:46.548571 10628 heartbeater.cc:347] Connected to a master server at 127.29.3.124:32799
I20260811 22:20:46.548858 10628 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.549402 10628 heartbeater.cc:511] Master 127.29.3.124:32799 requested a full tablet report, sending...
I20260811 22:20:46.550676 9822 ts_manager.cc:194] Registered new tserver with Master: d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707)
I20260811 22:20:46.548767 10633 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.551318 10633 heartbeater.cc:511] Master 127.29.3.125:44223 requested a full tablet report, sending...
I20260811 22:20:46.548767 10630 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:46.551656 10630 heartbeater.cc:511] Master 127.29.3.126:39659 requested a full tablet report, sending...
I20260811 22:20:46.552338 9515 ts_manager.cc:194] Registered new tserver with Master: d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707)
I20260811 22:20:46.553226 8464 ts_manager.cc:194] Re-registered known tserver with Master: d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707)
I20260811 22:20:46.553923 8464 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:39877
I20260811 22:20:46.554656 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.542246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46218 (local address 127.29.3.126:39659)
0811 22:20:46.543328 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.543332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.543683 (+ 351us) server_negotiation.cc:407] Connection header received
0811 22:20:46.543722 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.543725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.543772 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.543849 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.544838 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.545625 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.547891 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.548197 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.548289 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.549838 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.549842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.549844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.549869 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.551905 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.552457 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.552460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.552462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.552555 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.552864 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.552867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.552869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.553074 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.553195 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.554388 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.554509 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":105,"threads_started":1}
I20260811 22:20:46.558260 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.542695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.29.3.125:44223)
0811 22:20:46.546821 (+ 4126us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.546825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.549847 (+ 3022us) server_negotiation.cc:407] Connection header received
0811 22:20:46.549927 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.549930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.549971 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.550042 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.550875 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.551664 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.552917 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.553247 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.553309 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.555117 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.555120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.555122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.555143 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.556558 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.557036 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.557040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.557042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.557125 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.557411 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.557414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.557416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.557634 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.557751 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.557993 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.558089 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4072,"thread_start_us":96,"threads_started":1}
I20260811 22:20:46.559823 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.550032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.29.3.124:32799)
0811 22:20:46.550139 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.550143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.550154 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:20:46.550181 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.550183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.550221 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.550310 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.551047 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.551764 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.552917 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.553241 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.553309 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.556713 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.556717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.556720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.556743 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.558195 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.558635 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.558638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.558640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.558718 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.558981 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.558985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.558987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.559211 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.559326 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.559566 (+ 240us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.559671 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:20:46.558471 8464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:46.561951 8464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:46.564478 10658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: NON_VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: true } } }
I20260811 22:20:46.564620 10658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:46.568195 9772 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: attempting to promote NON_VOTER 0db3c0a21e4b48cd9a63a1723f6eb85a to VOTER
I20260811 22:20:46.568481 9772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:46.569934 9841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 LEARNER]: Refusing update from remote peer 4920be7096e146b48b848d87c4371cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 22:20:46.570199 9529 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Refusing update from remote peer 4920be7096e146b48b848d87c4371cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 22:20:46.570340 10658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 22:20:46.570787 10677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 22:20:46.572384 10658 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.572846 9841 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.573729 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.573828 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:46.573936 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.574023 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:46.574127 9772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.574250 9772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:46.574522 9888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.574630 9888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:46.574864 9529 raft_consensus.cc:2932] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.578063 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4920be7096e146b48b848d87c4371cdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:46.578192 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:46.581305 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.575446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:46393 (local address 127.0.0.1:40776)
0811 22:20:46.575895 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.575910 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.575988 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.576908 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.576912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.576933 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:46.577041 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.577048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.578588 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.578591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.579767 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.579779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.580915 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.580919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.581020 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.581048 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.581099 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.581169 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":26}
I20260811 22:20:46.581648 10683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.576180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40707 (local address 127.0.0.1:49882)
0811 22:20:46.576739 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.576752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.576833 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.577607 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.577609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.577625 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:46.577694 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.577698 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.578918 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.578922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.580200 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.580211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.581263 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.581266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.581356 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.581382 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.581452 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.581520 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":99,"threads_started":1}
I20260811 22:20:46.581827 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.576590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.29.3.67:40707)
0811 22:20:46.577009 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.577021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.577044 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:20:46.577117 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.577122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.577313 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.577463 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:46.577808 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.578792 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.580333 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.581121 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.581226 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.581372 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.581497 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.581590 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":96,"threads_started":1}
I20260811 22:20:46.582322 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.575966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:33273 (local address 127.0.0.1:47942)
0811 22:20:46.576233 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.576246 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.576332 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.578882 (+ 2550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.578885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.578902 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:46.579035 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.579042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.580064 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.580068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.580934 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.580945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.581906 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.581909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.581993 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.582013 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.582087 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.582154 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":22,"thread_start_us":116,"threads_started":1}
I20260811 22:20:46.582358 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.576609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.29.3.65:33273)
0811 22:20:46.578477 (+ 1868us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.578494 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.578516 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:20:46.578590 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.578595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.578753 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.578881 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:46.579148 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.579940 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.581063 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.581796 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.581893 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.582031 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.582111 (+ 80us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.582192 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":68,"threads_started":1}
I20260811 22:20:46.582813 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.575631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.29.3.66:46393)
0811 22:20:46.576377 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.576390 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.576417 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:20:46.576501 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.576506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.576737 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.577205 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:46.577363 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.578465 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.579899 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.580779 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.581844 (+ 1065us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.582127 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.582176 (+ 49us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.582261 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":129,"threads_started":1}
I20260811 22:20:46.584740 10501 tablet_service.cc:1467] Processing CreateTablet for tablet 6380d2580e284511ae12735c7c642f4a (DEFAULT_TABLE table=test-table [id=83e1947fbb164394af39d7bee36d0703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:46.584956 10021 tablet_service.cc:1467] Processing CreateTablet for tablet 6380d2580e284511ae12735c7c642f4a (DEFAULT_TABLE table=test-table [id=83e1947fbb164394af39d7bee36d0703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:46.585841 10501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6380d2580e284511ae12735c7c642f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:46.585966 10021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6380d2580e284511ae12735c7c642f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:46.587740 10268 tablet_service.cc:1467] Processing CreateTablet for tablet 6380d2580e284511ae12735c7c642f4a (DEFAULT_TABLE table=test-table [id=83e1947fbb164394af39d7bee36d0703]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:46.588757 10268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6380d2580e284511ae12735c7c642f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:46.594385 10695 tablet_bootstrap.cc:492] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Bootstrap starting.
I20260811 22:20:46.596971 10695 tablet_bootstrap.cc:654] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:46.597092 10699 tablet_bootstrap.cc:492] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Bootstrap starting.
I20260811 22:20:46.598001 10695 log.cc:824] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:46.598085 10697 tablet_bootstrap.cc:492] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Bootstrap starting.
I20260811 22:20:46.599918 10699 tablet_bootstrap.cc:654] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:46.600204 10695 tablet_bootstrap.cc:492] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: No bootstrap required, opened a new log
I20260811 22:20:46.600402 10695 ts_tablet_manager.cc:1397] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260811 22:20:46.600872 10697 tablet_bootstrap.cc:654] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:46.601181 10699 log.cc:824] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:46.601873 10697 log.cc:824] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:46.603499 10699 tablet_bootstrap.cc:492] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: No bootstrap required, opened a new log
I20260811 22:20:46.603701 10699 ts_tablet_manager.cc:1397] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:20:46.603912 10697 tablet_bootstrap.cc:492] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: No bootstrap required, opened a new log
I20260811 22:20:46.604095 10697 ts_tablet_manager.cc:1397] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:20:46.605165 10695 raft_consensus.cc:348] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.605463 10695 raft_consensus.cc:374] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:46.605535 10695 raft_consensus.cc:729] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed735695b25402aa239b0452e6ce7e6, State: Initialized, Role: FOLLOWER
I20260811 22:20:46.606067 10695 consensus_queue.cc:260] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.607227 10695 ts_tablet_manager.cc:1428] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260811 22:20:46.608798 10138 heartbeater.cc:503] Master 127.29.3.126:39659 was elected leader, sending a full tablet report...
I20260811 22:20:46.608639 10699 raft_consensus.cc:348] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.609436 10699 raft_consensus.cc:374] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:46.609493 10699 raft_consensus.cc:729] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345fc1b58590433a8cb2eca25e4deec7, State: Initialized, Role: FOLLOWER
I20260811 22:20:46.609951 10699 consensus_queue.cc:260] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.610975 10699 ts_tablet_manager.cc:1428] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260811 22:20:46.612578 10382 heartbeater.cc:503] Master 127.29.3.126:39659 was elected leader, sending a full tablet report...
I20260811 22:20:46.612988 10697 raft_consensus.cc:348] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.613276 10697 raft_consensus.cc:374] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:46.613360 10697 raft_consensus.cc:729] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8dbab3c846c4825b4f0315139d4c099, State: Initialized, Role: FOLLOWER
I20260811 22:20:46.613831 10697 consensus_queue.cc:260] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.615604 10697 ts_tablet_manager.cc:1428] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260811 22:20:46.615871 10630 heartbeater.cc:503] Master 127.29.3.126:39659 was elected leader, sending a full tablet report...
I20260811 22:20:46.641943 10712 raft_consensus.cc:482] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:46.642191 10712 raft_consensus.cc:504] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.643388 10712 leader_election.cc:288] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707), aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:46.647121 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.643594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40707 (local address 127.29.3.66:50089)
0811 22:20:46.643714 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.643730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.643815 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.644196 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.644199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.644231 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:46.644370 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.644375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.645199 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.645203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.646111 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.646121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.646750 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.646754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.646847 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.646873 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.646935 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.647003 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":19,"spinlock_wait_cycles":5120}
I20260811 22:20:46.647178 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.643827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:50089 (local address 127.29.3.67:40707)
0811 22:20:46.643984 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.643987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.643999 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:46.644053 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.644055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.644096 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.644160 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:46.644481 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.645097 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.646224 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.646638 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.646722 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.646787 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.646989 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.647058 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 22:20:46.647243 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.643685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:33273 (local address 127.29.3.66:34461)
0811 22:20:46.643803 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:46.643817 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:46.643896 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:46.644194 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:46.644197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:46.644223 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:46.644352 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.644357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.645150 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.645153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.646076 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:46.646086 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.646787 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.646790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:46.646895 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.646913 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 22:20:46.646982 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 22:20:46.647052 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":5,"spinlock_wait_cycles":2816}
I20260811 22:20:46.647377 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.643884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:34461 (local address 127.29.3.65:33273)
0811 22:20:46.644009 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.644012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.644024 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:46.644066 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.644068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.644107 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.644169 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:46.644454 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.645024 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.646199 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.646662 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.646767 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.646829 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.647191 (+ 362us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.647268 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:20:46.647948 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380d2580e284511ae12735c7c642f4a" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed735695b25402aa239b0452e6ce7e6" is_pre_election: true
I20260811 22:20:46.647985 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380d2580e284511ae12735c7c642f4a" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dbab3c846c4825b4f0315139d4c099" is_pre_election: true
I20260811 22:20:46.648295 10054 raft_consensus.cc:2454] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 0.
I20260811 22:20:46.648295 10535 raft_consensus.cc:2454] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 0.
I20260811 22:20:46.648867 10177 leader_election.cc:302] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345fc1b58590433a8cb2eca25e4deec7, d8dbab3c846c4825b4f0315139d4c099; no voters:
I20260811 22:20:46.649137 10712 raft_consensus.cc:2781] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:46.649222 10712 raft_consensus.cc:482] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:46.649267 10712 raft_consensus.cc:3037] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:46.650477 10712 raft_consensus.cc:504] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.650831 10712 leader_election.cc:288] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 election: Requested vote from peers d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707), aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:46.651252 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380d2580e284511ae12735c7c642f4a" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed735695b25402aa239b0452e6ce7e6"
I20260811 22:20:46.651185 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6380d2580e284511ae12735c7c642f4a" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dbab3c846c4825b4f0315139d4c099"
I20260811 22:20:46.651373 10054 raft_consensus.cc:3037] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:46.651399 10535 raft_consensus.cc:3037] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:46.652511 10054 raft_consensus.cc:2454] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 1.
I20260811 22:20:46.652763 10535 raft_consensus.cc:2454] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 1.
I20260811 22:20:46.652834 10176 leader_election.cc:302] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345fc1b58590433a8cb2eca25e4deec7, aed735695b25402aa239b0452e6ce7e6; no voters:
I20260811 22:20:46.653005 10712 raft_consensus.cc:2781] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:46.653218 10712 raft_consensus.cc:686] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 1 LEADER]: Becoming Leader. State: Replica: 345fc1b58590433a8cb2eca25e4deec7, State: Running, Role: LEADER
I20260811 22:20:46.653719 10712 consensus_queue.cc:237] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:46.657168 8463 catalog_manager.cc:5495] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66). New cstate: current_term: 1 leader_uuid: "345fc1b58590433a8cb2eca25e4deec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:46.670578 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.666544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.29.3.66:46393)
0811 22:20:46.667182 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.667185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.667196 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:20:46.667257 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.667259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.667309 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.667388 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:46.667777 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.668284 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.669069 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.669293 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.669371 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.669728 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:46.670133 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:46.670222 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.670266 (+ 44us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.670358 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566}
I20260811 22:20:46.676848 10535 raft_consensus.cc:1261] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Refusing update from remote peer 345fc1b58590433a8cb2eca25e4deec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:46.676858 10054 raft_consensus.cc:1261] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 1 FOLLOWER]: Refusing update from remote peer 345fc1b58590433a8cb2eca25e4deec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:46.677506 10712 consensus_queue.cc:1035] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:46.677734 10714 consensus_queue.cc:1035] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:46.685477 10719 mvcc.cc:205] Tried to move back new op lower bound from 7317450123981357056 to 7317450123896348672. Current Snapshot: MvccSnapshot[applied={T|T < 7317450123981357056}]
I20260811 22:20:46.688355 10721 mvcc.cc:205] Tried to move back new op lower bound from 7317450123981357056 to 7317450123896348672. Current Snapshot: MvccSnapshot[applied={T|T < 7317450123981357056}]
I20260811 22:20:46.696555 8463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:46.696709 8463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:46.701719 8463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4920be7096e146b48b848d87c4371cdb: Sending DeleteTablet for 3 replicas of tablet 6380d2580e284511ae12735c7c642f4a
I20260811 22:20:46.702538 10268 tablet_service.cc:1514] Processing DeleteTablet for tablet 6380d2580e284511ae12735c7c642f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:46 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40776
I20260811 22:20:46.702648 10021 tablet_service.cc:1514] Processing DeleteTablet for tablet 6380d2580e284511ae12735c7c642f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:46 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:47942
I20260811 22:20:46.703264 10733 tablet_replica.cc:330] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: stopping tablet replica
I20260811 22:20:46.703667 10733 raft_consensus.cc:2229] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:46.704165 10733 raft_consensus.cc:2258] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:46.704180 10501 tablet_service.cc:1514] Processing DeleteTablet for tablet 6380d2580e284511ae12735c7c642f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:46 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:49882
I20260811 22:20:46.705755 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 8408
I20260811 22:20:46.711496 10732 tablet_replica.cc:330] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: stopping tablet replica
I20260811 22:20:46.711854 10732 raft_consensus.cc:2229] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:46.712198 10733 ts_tablet_manager.cc:1905] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:46.712584 10732 raft_consensus.cc:2258] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:46.715075 10733 ts_tablet_manager.cc:1918] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:46.717677 10732 ts_tablet_manager.cc:1905] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:46.720633 10732 ts_tablet_manager.cc:1918] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:46.720844 10733 log.cc:1196] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/6380d2580e284511ae12735c7c642f4a
I20260811 22:20:46.721299 10733 ts_tablet_manager.cc:1939] T 6380d2580e284511ae12735c7c642f4a P aed735695b25402aa239b0452e6ce7e6: Deleting consensus metadata
I20260811 22:20:46.722915 10732 log.cc:1196] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/6380d2580e284511ae12735c7c642f4a
I20260811 22:20:46.723361 10732 ts_tablet_manager.cc:1939] T 6380d2580e284511ae12735c7c642f4a P 345fc1b58590433a8cb2eca25e4deec7: Deleting consensus metadata
I20260811 22:20:46.723558 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.703524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.29.3.125:44223)
0811 22:20:46.703630 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.703634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.703901 (+ 267us) server_negotiation.cc:407] Connection header received
0811 22:20:46.704093 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.704096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.704143 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.704193 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.705791 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.706503 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.717571 (+ 11068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.717896 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.717979 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.719901 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.719905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.719908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.719937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.721608 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.722110 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.722114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.722116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.722205 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.722549 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.722552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.722553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.722784 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.722883 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.723277 (+ 394us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.723402 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:20:46.725059 10737 tablet_replica.cc:330] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: stopping tablet replica
I20260811 22:20:46.725431 10737 raft_consensus.cc:2229] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:46.725775 10737 raft_consensus.cc:2258] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 22:20:46.730223 9938 connection.cc:535] server connection from 127.0.0.1:47942 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 22:20:46.730808 10177 connection.cc:535] server connection from 127.0.0.1:40776 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 22:20:46.731595 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.705148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35498 (local address 127.29.3.124:32799)
0811 22:20:46.705278 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.705282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.724184 (+ 18902us) server_negotiation.cc:407] Connection header received
0811 22:20:46.724377 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.724381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.724436 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.724487 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.725319 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.725884 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.726729 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.726942 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.727031 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.728219 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.728222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.728224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.728248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.729477 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.730031 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.730035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.730037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.730131 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.730593 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.730596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.730598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.730845 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.730971 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.731305 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.731427 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:20:46.733260 10737 ts_tablet_manager.cc:1905] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:46.736070 10737 ts_tablet_manager.cc:1918] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:46.736184 10737 log.cc:1196] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/6380d2580e284511ae12735c7c642f4a
I20260811 22:20:46.736509 10737 ts_tablet_manager.cc:1939] T 6380d2580e284511ae12735c7c642f4a P d8dbab3c846c4825b4f0315139d4c099: Deleting consensus metadata
W20260811 22:20:46.737700 10431 connection.cc:441] server connection from 127.0.0.1:49882 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49882 (request call id 1) could send its response
I20260811 22:20:46.744438 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.734518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.29.3.125:44223)
0811 22:20:46.734645 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.734649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.735201 (+ 552us) server_negotiation.cc:407] Connection header received
0811 22:20:46.735405 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.735408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.735464 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.735582 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.736689 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.737419 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.738372 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.738598 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.738693 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.740398 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.740401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.740403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.740426 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.742104 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.742600 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.742604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.742606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.742687 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.742934 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.742938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.742939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.743182 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.743312 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.744162 (+ 850us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.744281 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:20:46.744673 10144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:46.734957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.29.3.124:32799)
0811 22:20:46.735062 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:46.735066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:46.735875 (+ 809us) server_negotiation.cc:407] Connection header received
0811 22:20:46.736090 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:46.736093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:46.736137 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:46.736182 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:46.737287 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.738061 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.739413 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:46.739763 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:46.739886 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:46.741328 (+ 1442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:46.741332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:46.741335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:46.741363 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:46.742884 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.743275 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.743279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.743281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.743365 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:46.743730 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:46.743734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:46.743736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:46.743944 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:46.744077 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:46.744306 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 22:20:46.744410 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:20:47.689603 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:47.689038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:42525)
0811 22:20:47.689382 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:47.689487 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":80,"threads_started":1}
W20260811 22:20:47.689951 10630 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:39659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:47.690542 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:47.690136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:48337)
0811 22:20:47.690377 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:47.690462 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
W20260811 22:20:47.690838 10382 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:39659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:47.693543 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:47.693188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:44293)
0811 22:20:47.693393 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:47.693466 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
W20260811 22:20:47.693804 10138 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:39659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:48.201347 10760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4920be7096e146b48b848d87c4371cdb)
I20260811 22:20:48.201581 10760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:48.202100 10762 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4920be7096e146b48b848d87c4371cdb)
I20260811 22:20:48.202277 10762 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:48.202728 10760 leader_election.cc:288] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659), aa74a57e54cf4161bd32e45f09aacdb2 (127.29.3.125:44223)
I20260811 22:20:48.203256 10762 leader_election.cc:288] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659), 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124:32799)
I20260811 22:20:48.203732 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.202870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43824)
0811 22:20:48.203194 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.203609 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":65,"threads_started":1}
I20260811 22:20:48.203797 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.203327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43834)
0811 22:20:48.203628 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.203701 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":70,"threads_started":1}
W20260811 22:20:48.203966 9485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
W20260811 22:20:48.204032 9804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:48.204985 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.204738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43836)
0811 22:20:48.204873 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.204918 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260811 22:20:48.205149 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.204900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43852)
0811 22:20:48.205034 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.205082 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260811 22:20:48.205149 9804 leader_election.cc:334] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
W20260811 22:20:48.205308 9485 leader_election.cc:334] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:48.207140 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.202957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:44223 (local address 127.0.0.1:48392)
0811 22:20:48.203284 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.203300 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:48.203409 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:48.203869 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:48.203872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:48.203894 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:48.204009 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:48.204016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.205004 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.205008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:48.205989 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:48.205999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.206723 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.206727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:48.206853 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:48.206872 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:20:48.206936 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:20:48.207012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":20,"thread_start_us":71,"threads_started":1}
I20260811 22:20:48.207137 10766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.203046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.29.3.125:44223)
0811 22:20:48.203611 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:48.203615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:48.203629 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:20:48.203675 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:48.203677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:48.203724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:48.203848 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:48.204141 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.204879 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.206105 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.206614 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.206713 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:48.206777 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:48.206961 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 22:20:48.207030 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":66,"threads_started":1}
I20260811 22:20:48.207644 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.203393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:32799 (local address 127.0.0.1:49412)
0811 22:20:48.204042 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.204055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:48.204143 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:48.204496 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:48.204499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:48.204516 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:48.204616 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:48.204623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.205542 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.205545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:48.206435 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:48.206445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.207101 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.207105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:48.207349 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:48.207373 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:20:48.207443 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:20:48.207517 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":43,"threads_started":1}
I20260811 22:20:48.207664 10769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.203558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49412 (local address 127.29.3.124:32799)
0811 22:20:48.203789 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:48.203793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:48.204232 (+ 439us) server_negotiation.cc:407] Connection header received
0811 22:20:48.204279 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:48.204282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:48.204323 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:48.204397 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:48.204736 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.205449 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.206547 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:48.206992 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:48.207140 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:48.207200 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:48.207489 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 22:20:48.207560 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260811 22:20:48.207619 9529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" is_pre_election: true
I20260811 22:20:48.207830 9529 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0db3c0a21e4b48cd9a63a1723f6eb85a in term 1.
I20260811 22:20:48.208149 9841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" is_pre_election: true
I20260811 22:20:48.208180 9807 leader_election.cc:302] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0db3c0a21e4b48cd9a63a1723f6eb85a, aa74a57e54cf4161bd32e45f09aacdb2; no voters: 4920be7096e146b48b848d87c4371cdb
I20260811 22:20:48.208329 9841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa74a57e54cf4161bd32e45f09aacdb2 in term 1.
I20260811 22:20:48.208442 10760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 22:20:48.208511 10760 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4920be7096e146b48b848d87c4371cdb)
I20260811 22:20:48.208560 10760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:20:48.208742 9489 leader_election.cc:302] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0db3c0a21e4b48cd9a63a1723f6eb85a, aa74a57e54cf4161bd32e45f09aacdb2; no voters: 4920be7096e146b48b848d87c4371cdb
I20260811 22:20:48.208952 10762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 22:20:48.209008 10762 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4920be7096e146b48b848d87c4371cdb)
I20260811 22:20:48.209050 10762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:20:48.209765 10760 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:48.210037 10762 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:48.210119 10760 leader_election.cc:288] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [CANDIDATE]: Term 2 election: Requested vote from peers 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659), aa74a57e54cf4161bd32e45f09aacdb2 (127.29.3.125:44223)
I20260811 22:20:48.210419 10762 leader_election.cc:288] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 2 election: Requested vote from peers 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659), 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124:32799)
I20260811 22:20:48.210752 9529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "aa74a57e54cf4161bd32e45f09aacdb2"
I20260811 22:20:48.210737 9841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
I20260811 22:20:48.210817 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.210606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43882)
0811 22:20:48.210719 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.210767 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 22:20:48.210906 9841 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate aa74a57e54cf4161bd32e45f09aacdb2 in current term 2: Already voted for candidate 0db3c0a21e4b48cd9a63a1723f6eb85a in this term.
I20260811 22:20:48.210906 9529 raft_consensus.cc:2379] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0db3c0a21e4b48cd9a63a1723f6eb85a in current term 2: Already voted for candidate aa74a57e54cf4161bd32e45f09aacdb2 in this term.
I20260811 22:20:48.211557 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.210232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43868)
0811 22:20:48.211439 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.211492 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1144}
I20260811 22:20:48.211567 10767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.211383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43892)
0811 22:20:48.211477 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.211519 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:20:48.211786 9485 leader_election.cc:334] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:48.211860 9485 leader_election.cc:302] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa74a57e54cf4161bd32e45f09aacdb2; no voters: 0db3c0a21e4b48cd9a63a1723f6eb85a, 4920be7096e146b48b848d87c4371cdb
I20260811 22:20:48.211993 10762 raft_consensus.cc:2726] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 22:20:48.212190 10764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.211981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:43904)
0811 22:20:48.212082 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.212134 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 22:20:48.212260 9804 leader_election.cc:334] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:48.212344 9804 leader_election.cc:302] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0db3c0a21e4b48cd9a63a1723f6eb85a; no voters: 4920be7096e146b48b848d87c4371cdb, aa74a57e54cf4161bd32e45f09aacdb2
I20260811 22:20:48.212472 10760 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260811 22:20:48.691390 10772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.690655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:46203)
0811 22:20:48.691015 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.691273 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":101,"threads_started":1}
I20260811 22:20:48.691848 10775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.691332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:35875)
0811 22:20:48.691668 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.691754 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":87,"threads_started":1}
I20260811 22:20:48.694623 10778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:48.694237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:51779)
0811 22:20:48.694494 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:48.694547 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
I20260811 22:20:49.692842 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.692195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:49071)
0811 22:20:49.692651 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.692726 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":82,"threads_started":1}
I20260811 22:20:49.693315 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.692837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:55003)
0811 22:20:49.693171 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.693227 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":67,"threads_started":1}
I20260811 22:20:49.695502 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.695138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:35223)
0811 22:20:49.695372 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.695426 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":61,"threads_started":1}
I20260811 22:20:49.988946 10787 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:49.989087 10787 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:49.989535 10787 leader_election.cc:288] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659), 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124:32799)
I20260811 22:20:49.989987 9841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" is_pre_election: true
I20260811 22:20:49.990077 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.989727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:44146)
0811 22:20:49.989933 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.989992 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":56,"threads_started":1}
I20260811 22:20:49.990134 9841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa74a57e54cf4161bd32e45f09aacdb2 in term 2.
I20260811 22:20:49.990458 9489 leader_election.cc:302] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0db3c0a21e4b48cd9a63a1723f6eb85a, aa74a57e54cf4161bd32e45f09aacdb2; no voters:
I20260811 22:20:49.990640 10787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260811 22:20:49.990727 10787 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:49.990777 10787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 2 FOLLOWER]: Advancing to term 3
I20260811 22:20:49.990950 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.990721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:44162)
0811 22:20:49.990829 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.990895 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 22:20:49.991055 9485 leader_election.cc:334] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:49.991863 10787 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:49.992169 10787 leader_election.cc:288] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 3 election: Requested vote from peers 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659), 0db3c0a21e4b48cd9a63a1723f6eb85a (127.29.3.124:32799)
I20260811 22:20:49.992501 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.992293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:44170)
0811 22:20:49.992390 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.992441 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 22:20:49.992640 9841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a"
I20260811 22:20:49.992758 9841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 2 FOLLOWER]: Advancing to term 3
I20260811 22:20:49.993139 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:49.992909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:44174)
0811 22:20:49.993029 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:49.993082 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 22:20:49.993264 9485 leader_election.cc:334] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
I20260811 22:20:49.993839 9841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa74a57e54cf4161bd32e45f09aacdb2 in term 3.
I20260811 22:20:49.994105 9489 leader_election.cc:302] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0db3c0a21e4b48cd9a63a1723f6eb85a, aa74a57e54cf4161bd32e45f09aacdb2; no voters: 4920be7096e146b48b848d87c4371cdb
I20260811 22:20:49.994251 10787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 22:20:49.994447 10787 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [term 3 LEADER]: Becoming Leader. State: Replica: aa74a57e54cf4161bd32e45f09aacdb2, State: Running, Role: LEADER
I20260811 22:20:49.994704 10787 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } }
I20260811 22:20:49.995710 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa74a57e54cf4161bd32e45f09aacdb2. Latest consensus state: current_term: 3 leader_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:49.995823 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:49.996104 10794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:49.998489 10794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:49.998890 10794 catalog_manager.cc:1259] Loaded cluster ID: d1e47123fc144161a2424f9e06620014
I20260811 22:20:49.998936 10794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:49.999495 10794 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:49.999871 10794 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Loaded TSK: 0
I20260811 22:20:50.000638 10794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:20:50.034660 9515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:50.037911 9515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:50.041010 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.040689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:44186)
0811 22:20:50.040845 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.040922 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 22:20:50.041251 9841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [term 3 FOLLOWER]: Refusing update from remote peer aa74a57e54cf4161bd32e45f09aacdb2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260811 22:20:50.041759 10790 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260811 22:20:50.041983 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.041748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.0.0.1:44194)
0811 22:20:50.041865 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.041916 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 22:20:50.042093 9485 consensus_peers.cc:487] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 -> Peer 4920be7096e146b48b848d87c4371cdb (127.29.3.126:39659): Couldn't send request to peer 4920be7096e146b48b848d87c4371cdb. Status: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 22:20:50.043192 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa74a57e54cf4161bd32e45f09aacdb2. Latest consensus state: current_term: 3 leader_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:50.043318 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:50.044315 10790 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:50.044430 10790 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:50.044883 10798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4920be7096e146b48b848d87c4371cdb" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39659 } } peers { permanent_uuid: "aa74a57e54cf4161bd32e45f09aacdb2" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 44223 } attrs { promote: false } } peers { permanent_uuid: "0db3c0a21e4b48cd9a63a1723f6eb85a" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 32799 } attrs { promote: false } } }
I20260811 22:20:50.044998 10798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0db3c0a21e4b48cd9a63a1723f6eb85a [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:20:50.054378 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.050080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40707 (local address 127.0.0.1:40178)
0811 22:20:50.050226 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.050241 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:50.050367 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:50.050812 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:50.050815 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:50.050834 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:50.050943 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:50.050950 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.051903 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.051906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:50.053054 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:50.053063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.053949 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.053952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:50.054042 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.054069 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 22:20:50.054143 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 22:20:50.054216 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 22:20:50.054411 10807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.050208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.29.3.67:40707)
0811 22:20:50.050577 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:50.050582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:50.050597 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:50.050653 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:50.050656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:50.050706 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:50.050792 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:50.051050 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.051807 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.053183 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.053834 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.053940 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.054008 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:50.054169 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 22:20:50.054239 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":123,"threads_started":1}
I20260811 22:20:50.055011 10501 tablet_service.cc:1467] Processing CreateTablet for tablet 45eb5584d1b54ed381a52a9e39896662 (DEFAULT_TABLE table=test-table [id=a4e3a65abd554148b5893480080fb47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:50.055205 10811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.050314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:46393 (local address 127.0.0.1:39220)
0811 22:20:50.051105 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.051118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:50.051206 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:50.051493 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:50.051495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:50.051511 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:50.051582 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:50.051587 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.052354 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.052357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:50.053155 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:50.053167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.054833 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.054837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:50.054922 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.054943 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:20:50.055006 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 22:20:50.055069 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":21,"thread_start_us":115,"threads_started":1}
I20260811 22:20:50.055226 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.051020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39220 (local address 127.29.3.66:46393)
0811 22:20:50.051244 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:50.051248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:50.051261 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:50.051311 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:50.051313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:50.051356 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:50.051433 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:50.051704 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.052231 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.053281 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.053849 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.054837 (+ 988us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.054898 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:50.055040 (+ 142us) server_negotiation.cc:299] Negotiation successful
0811 22:20:50.055111 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260811 22:20:50.055347 10501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eb5584d1b54ed381a52a9e39896662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:50.055727 10268 tablet_service.cc:1467] Processing CreateTablet for tablet 45eb5584d1b54ed381a52a9e39896662 (DEFAULT_TABLE table=test-table [id=a4e3a65abd554148b5893480080fb47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:50.055976 10268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eb5584d1b54ed381a52a9e39896662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:50.057999 10815 tablet_bootstrap.cc:492] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Bootstrap starting.
I20260811 22:20:50.058681 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.050665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:33273 (local address 127.0.0.1:55272)
0811 22:20:50.051616 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.051639 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:50.051712 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:50.052258 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:50.052261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:50.052278 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:50.052417 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:50.052423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.053349 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.053352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:50.054425 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:50.054437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.055302 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.055305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:50.055398 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.055419 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:20:50.058475 (+ 3056us) client_negotiation.cc:240] Negotiation successful
0811 22:20:50.058554 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":22,"thread_start_us":94,"threads_started":1}
I20260811 22:20:50.058851 10812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.050787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.29.3.65:33273)
0811 22:20:50.051958 (+ 1171us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:50.051962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:50.051975 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:50.052022 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:50.052025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:50.052070 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:50.052157 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:50.052542 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.053262 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.054563 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.055082 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.055182 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.055239 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:50.058598 (+ 3359us) server_negotiation.cc:299] Negotiation successful
0811 22:20:50.058721 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":104,"threads_started":1}
I20260811 22:20:50.059216 10815 tablet_bootstrap.cc:654] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:50.059391 10021 tablet_service.cc:1467] Processing CreateTablet for tablet 45eb5584d1b54ed381a52a9e39896662 (DEFAULT_TABLE table=test-table [id=a4e3a65abd554148b5893480080fb47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:50.059653 10021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45eb5584d1b54ed381a52a9e39896662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:50.059834 10817 tablet_bootstrap.cc:492] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Bootstrap starting.
I20260811 22:20:50.060355 10815 tablet_bootstrap.cc:492] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: No bootstrap required, opened a new log
I20260811 22:20:50.060429 10815 ts_tablet_manager.cc:1397] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:50.060856 10815 raft_consensus.cc:348] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.060977 10815 raft_consensus.cc:374] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:50.061017 10815 raft_consensus.cc:729] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8dbab3c846c4825b4f0315139d4c099, State: Initialized, Role: FOLLOWER
I20260811 22:20:50.061179 10815 consensus_queue.cc:260] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.061537 10815 ts_tablet_manager.cc:1428] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:50.062107 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.061887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:57841)
0811 22:20:50.061983 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.062039 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260811 22:20:50.062789 10817 tablet_bootstrap.cc:654] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:50.063261 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.063058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:46625)
0811 22:20:50.063147 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.063205 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260811 22:20:50.063787 10822 tablet_bootstrap.cc:492] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6: Bootstrap starting.
I20260811 22:20:50.064018 10817 tablet_bootstrap.cc:492] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: No bootstrap required, opened a new log
I20260811 22:20:50.064109 10817 ts_tablet_manager.cc:1397] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260811 22:20:50.064265 10633 heartbeater.cc:503] Master 127.29.3.125:44223 was elected leader, sending a full tablet report...
I20260811 22:20:50.064517 10817 raft_consensus.cc:348] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.064631 10817 raft_consensus.cc:374] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:50.064682 10817 raft_consensus.cc:729] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345fc1b58590433a8cb2eca25e4deec7, State: Initialized, Role: FOLLOWER
I20260811 22:20:50.064831 10817 consensus_queue.cc:260] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.065033 10822 tablet_bootstrap.cc:654] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:50.065464 10817 ts_tablet_manager.cc:1428] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:50.066684 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.066447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:37663)
0811 22:20:50.066556 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.066615 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260811 22:20:50.066699 10822 tablet_bootstrap.cc:492] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6: No bootstrap required, opened a new log
I20260811 22:20:50.066795 10822 ts_tablet_manager.cc:1397] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:50.067245 10822 raft_consensus.cc:348] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.067363 10822 raft_consensus.cc:374] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:50.067397 10822 raft_consensus.cc:729] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed735695b25402aa239b0452e6ce7e6, State: Initialized, Role: FOLLOWER
I20260811 22:20:50.067518 10822 consensus_queue.cc:260] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.067853 10822 ts_tablet_manager.cc:1428] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:50.068849 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.068397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:48141)
0811 22:20:50.068731 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.068786 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271}
I20260811 22:20:50.069190 10139 heartbeater.cc:503] Master 127.29.3.125:44223 was elected leader, sending a full tablet report...
I20260811 22:20:50.069619 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.069390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:52655)
0811 22:20:50.069501 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.069556 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 22:20:50.073523 10825 raft_consensus.cc:482] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:50.073629 10825 raft_consensus.cc:504] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.073953 10825 leader_election.cc:288] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707), aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:50.074307 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb5584d1b54ed381a52a9e39896662" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed735695b25402aa239b0452e6ce7e6" is_pre_election: true
I20260811 22:20:50.074352 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb5584d1b54ed381a52a9e39896662" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dbab3c846c4825b4f0315139d4c099" is_pre_election: true
I20260811 22:20:50.074441 10054 raft_consensus.cc:2454] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 0.
I20260811 22:20:50.074476 10535 raft_consensus.cc:2454] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 0.
I20260811 22:20:50.074757 10176 leader_election.cc:302] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345fc1b58590433a8cb2eca25e4deec7, aed735695b25402aa239b0452e6ce7e6; no voters:
I20260811 22:20:50.074928 10825 raft_consensus.cc:2781] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:50.074990 10825 raft_consensus.cc:482] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:50.075021 10825 raft_consensus.cc:3037] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:50.075904 10825 raft_consensus.cc:504] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.076192 10825 leader_election.cc:288] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 election: Requested vote from peers d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707), aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273)
I20260811 22:20:50.076491 10535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb5584d1b54ed381a52a9e39896662" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8dbab3c846c4825b4f0315139d4c099"
I20260811 22:20:50.076491 10054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45eb5584d1b54ed381a52a9e39896662" candidate_uuid: "345fc1b58590433a8cb2eca25e4deec7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed735695b25402aa239b0452e6ce7e6"
I20260811 22:20:50.076606 10054 raft_consensus.cc:3037] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:50.076606 10535 raft_consensus.cc:3037] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 0 FOLLOWER]: Advancing to term 1
W20260811 22:20:50.077381 10384 tablet.cc:2369] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:50.077654 10535 raft_consensus.cc:2454] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 1.
I20260811 22:20:50.077654 10054 raft_consensus.cc:2454] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 345fc1b58590433a8cb2eca25e4deec7 in term 1.
I20260811 22:20:50.077934 10177 leader_election.cc:302] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 345fc1b58590433a8cb2eca25e4deec7, d8dbab3c846c4825b4f0315139d4c099; no voters:
I20260811 22:20:50.078115 10825 raft_consensus.cc:2781] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:50.078346 10825 raft_consensus.cc:686] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 1 LEADER]: Becoming Leader. State: Replica: 345fc1b58590433a8cb2eca25e4deec7, State: Running, Role: LEADER
I20260811 22:20:50.078486 10825 consensus_queue.cc:237] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } }
I20260811 22:20:50.078660 10383 heartbeater.cc:503] Master 127.29.3.125:44223 was elected leader, sending a full tablet report...
I20260811 22:20:50.079243 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.078964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:59457)
0811 22:20:50.079094 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.079170 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 22:20:50.080062 9515 catalog_manager.cc:5495] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66). New cstate: current_term: 1 leader_uuid: "345fc1b58590433a8cb2eca25e4deec7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "345fc1b58590433a8cb2eca25e4deec7" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:50.097064 10814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.094073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.29.3.66:46393)
0811 22:20:50.094198 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:50.094202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:50.094216 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:20:50.094339 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:50.094342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:50.094391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:50.094453 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:50.094653 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.095145 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.095995 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:50.096234 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:50.096324 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:50.096611 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:50.096684 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:50.096737 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:50.096868 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 22:20:50.096919 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:20:50.099251 10535 raft_consensus.cc:1261] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Refusing update from remote peer 345fc1b58590433a8cb2eca25e4deec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:50.099248 10054 raft_consensus.cc:1261] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6 [term 1 FOLLOWER]: Refusing update from remote peer 345fc1b58590433a8cb2eca25e4deec7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:50.099658 10825 consensus_queue.cc:1035] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed735695b25402aa239b0452e6ce7e6" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 33273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:50.099864 10830 consensus_queue.cc:1035] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8dbab3c846c4825b4f0315139d4c099" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:50.101481 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.101213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:46001)
0811 22:20:50.101336 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.101395 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 22:20:50.102254 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.102005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:45403)
0811 22:20:50.102147 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.102192 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260811 22:20:50.103195 10140 tablet.cc:2369] T 45eb5584d1b54ed381a52a9e39896662 P aed735695b25402aa239b0452e6ce7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:50.107858 10782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.107391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.65:57485)
0811 22:20:50.107728 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.107785 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260811 22:20:50.108608 10780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.108302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.66:55663)
0811 22:20:50.108474 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.108526 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260811 22:20:50.110013 10779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.109764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39659 (local address 127.29.3.67:57311)
0811 22:20:50.109895 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.109945 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 22:20:50.110368 9515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:50.110518 9515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:50.114212 9515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa74a57e54cf4161bd32e45f09aacdb2: Sending DeleteTablet for 3 replicas of tablet 45eb5584d1b54ed381a52a9e39896662
I20260811 22:20:50.114941 10501 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eb5584d1b54ed381a52a9e39896662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:50 UTC) from {username='oryx', principal='oryx/127.29.3.125@KRBTEST.COM'} at 127.0.0.1:40178
I20260811 22:20:50.115263 10268 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eb5584d1b54ed381a52a9e39896662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:50 UTC) from {username='oryx', principal='oryx/127.29.3.125@KRBTEST.COM'} at 127.0.0.1:39220
I20260811 22:20:50.115409 10860 tablet_replica.cc:330] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: stopping tablet replica
I20260811 22:20:50.115401 10021 tablet_service.cc:1514] Processing DeleteTablet for tablet 45eb5584d1b54ed381a52a9e39896662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:50 UTC) from {username='oryx', principal='oryx/127.29.3.125@KRBTEST.COM'} at 127.0.0.1:55272
I20260811 22:20:50.115520 10860 raft_consensus.cc:2229] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:50.115593 10862 tablet_replica.cc:330] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: stopping tablet replica
I20260811 22:20:50.115677 10860 raft_consensus.cc:2258] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:50.115705 10862 raft_consensus.cc:2229] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:50.115880 10862 raft_consensus.cc:2258] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:50.116086 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 9761
I20260811 22:20:50.116366 10860 ts_tablet_manager.cc:1905] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:50.118580 10860 ts_tablet_manager.cc:1918] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:50.118650 10860 log.cc:1196] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/45eb5584d1b54ed381a52a9e39896662
I20260811 22:20:50.118865 10860 ts_tablet_manager.cc:1939] T 45eb5584d1b54ed381a52a9e39896662 P d8dbab3c846c4825b4f0315139d4c099: Deleting consensus metadata
I20260811 22:20:50.119656 9488 catalog_manager.cc:4841] TS d8dbab3c846c4825b4f0315139d4c099 (127.29.3.67:40707): tablet 45eb5584d1b54ed381a52a9e39896662 (table test-table [id=a4e3a65abd554148b5893480080fb47c]) successfully deleted
I20260811 22:20:50.121572 10862 ts_tablet_manager.cc:1905] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:50.124627 10862 ts_tablet_manager.cc:1918] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:50.124708 10862 log.cc:1196] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/45eb5584d1b54ed381a52a9e39896662
I20260811 22:20:50.124944 10862 ts_tablet_manager.cc:1939] T 45eb5584d1b54ed381a52a9e39896662 P 345fc1b58590433a8cb2eca25e4deec7: Deleting consensus metadata
I20260811 22:20:50.125703 9485 catalog_manager.cc:4841] TS 345fc1b58590433a8cb2eca25e4deec7 (127.29.3.66:46393): tablet 45eb5584d1b54ed381a52a9e39896662 (table test-table [id=a4e3a65abd554148b5893480080fb47c]) successfully deleted
I20260811 22:20:50.140645 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 10146
I20260811 22:20:50.151443 10808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:50.140752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:33273 (local address 127.0.0.1:55274)
0811 22:20:50.151230 (+ 10478us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:50.151326 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:33273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10328}
W20260811 22:20:50.153450 9485 catalog_manager.cc:4584] TS aed735695b25402aa239b0452e6ce7e6 (127.29.3.65:33273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45eb5584d1b54ed381a52a9e39896662: Network error: Client connection negotiation failed: client connection to 127.29.3.65:33273: connect: Connection refused (error 111)
I20260811 22:20:50.161124 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 10393
I20260811 22:20:50.184839 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 9257
I20260811 22:20:50.198195 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 9461
2026-08-11T22:20:50Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7264 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:50 dist-test-slave-4tdj krb5kdc[10873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:50 dist-test-slave-4tdj krb5kdc[10873](info): set up 2 sockets
Aug 11 22:20:50 dist-test-slave-4tdj krb5kdc[10873](info): commencing operation
krb5kdc: starting...
W20260811 22:20:51.505139 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:51 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab.
Entry for principal oryx/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab.
I20260811 22:20:51.624935 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42757
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab
--principal=oryx/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:42757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:51.737751 10891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:51.738052 10891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:51.738098 10891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:51.742327 10891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:51.742401 10891 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 22:20:51.742419 10891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:51.742436 10891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:51.742465 10891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:51.742496 10891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:51.748016 10891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:42757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.126.keytab
--principal=oryx/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:42757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:51.749200 10891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:51.750365 10891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:51.758800 10904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:51.758781 10900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:51.758770 10902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:51.759131 10891 server_base.cc:1034] running on GCE node
I20260811 22:20:51.759739 10891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:51.760882 10891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:51.762039 10891 hybrid_clock.cc:648] HybridClock initialized: now 1786486851762023 us; error 30 us; skew 500 ppm
Aug 11 22:20:51 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486851, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:51.764824 10891 init.cc:377] Logged in from keytab as oryx/127.29.3.126@KRBTEST.COM (short username oryx)
I20260811 22:20:51.765756 10891 webserver.cc:460] Webserver started at http://127.29.3.126:35583/ using document root <none> and password file <none>
I20260811 22:20:51.766360 10891 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:51.766435 10891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:51.766646 10891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:51.768703 10891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "008e365a4be04ae0b0839ea402e04d70"
format_stamp: "Formatted at 2026-08-11 22:20:51 on dist-test-slave-4tdj"
server_key: "98433fce0b2f8d8f73fc03579a564923"
server_key_iv: "c8d93847789fdc3e01fc512a18317f69"
server_key_version: "encryptionkey@0"
I20260811 22:20:51.769269 10891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "008e365a4be04ae0b0839ea402e04d70"
format_stamp: "Formatted at 2026-08-11 22:20:51 on dist-test-slave-4tdj"
server_key: "98433fce0b2f8d8f73fc03579a564923"
server_key_iv: "c8d93847789fdc3e01fc512a18317f69"
server_key_version: "encryptionkey@0"
I20260811 22:20:51.773065 10891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:20:51.775487 10917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:51.776667 10891 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:51.776786 10891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "008e365a4be04ae0b0839ea402e04d70"
format_stamp: "Formatted at 2026-08-11 22:20:51 on dist-test-slave-4tdj"
server_key: "98433fce0b2f8d8f73fc03579a564923"
server_key_iv: "c8d93847789fdc3e01fc512a18317f69"
server_key_version: "encryptionkey@0"
I20260811 22:20:51.776880 10891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:51.789461 10891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:51.792519 10891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:51.792693 10891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:51.800180 10891 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:42757
I20260811 22:20:51.800246 11023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:42757 every 8 connection(s)
I20260811 22:20:51.801126 10891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:51.803977 11024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:51.809537 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Bootstrap starting.
I20260811 22:20:51.811018 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 10891
I20260811 22:20:51.811138 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:51.811399 29709 external_mini_cluster.cc:1428] Setting key b26915e42105a7a559d6297db07c6309
I20260811 22:20:51.812114 11024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:51.813019 11024 log.cc:824] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:51.814935 11024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: No bootstrap required, opened a new log
I20260811 22:20:51.818439 11024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008e365a4be04ae0b0839ea402e04d70" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42757 } }
I20260811 22:20:51.818670 11024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:51.818747 11024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008e365a4be04ae0b0839ea402e04d70, State: Initialized, Role: FOLLOWER
I20260811 22:20:51.819208 11024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008e365a4be04ae0b0839ea402e04d70" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42757 } }
I20260811 22:20:51.819376 11024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:51.819444 11024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:51.819553 11024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 22:20:51 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:51.820489 11024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008e365a4be04ae0b0839ea402e04d70" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42757 } }
I20260811 22:20:51.820856 11024 leader_election.cc:302] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 008e365a4be04ae0b0839ea402e04d70; no voters:
I20260811 22:20:51.821218 11024 leader_election.cc:288] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:51.821317 11030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:51.821547 11030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [term 1 LEADER]: Becoming Leader. State: Replica: 008e365a4be04ae0b0839ea402e04d70, State: Running, Role: LEADER
I20260811 22:20:51.821902 11030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008e365a4be04ae0b0839ea402e04d70" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42757 } }
I20260811 22:20:51.822553 11024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:51.823618 11031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "008e365a4be04ae0b0839ea402e04d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008e365a4be04ae0b0839ea402e04d70" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42757 } } }
I20260811 22:20:51.823772 11031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:51.823997 11034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 008e365a4be04ae0b0839ea402e04d70. Latest consensus state: current_term: 1 leader_uuid: "008e365a4be04ae0b0839ea402e04d70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008e365a4be04ae0b0839ea402e04d70" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 42757 } } }
I20260811 22:20:51.824108 11034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70 [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:51.824510 11045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:51.826406 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:51.812482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46830 (local address 127.29.3.126:42757)
0811 22:20:51.812906 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:51.812922 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:51.812956 (+ 34us) server_negotiation.cc:407] Connection header received
0811 22:20:51.813606 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:51.813625 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:51.813947 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:51.814263 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:51.815114 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:51.816077 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:51.816933 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:51.817430 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:51.817593 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:51.820343 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:51.820348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:51.820363 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:51.820390 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:51.822509 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:51.823120 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:51.823126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:51.823133 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:51.823301 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:51.823651 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:51.823654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:51.823656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:51.824194 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:51.824425 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:51.825256 (+ 831us) server_negotiation.cc:299] Negotiation successful
0811 22:20:51.825596 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":121,"threads_started":1}
I20260811 22:20:51.827808 11045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:51.833240 11045 catalog_manager.cc:1347] Generated new cluster ID: 7029ecfc286548b6a076340f9322aeac
I20260811 22:20:51.833309 11045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:51.859078 11045 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:51.860407 11045 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:51.867830 11045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Generated new TSK 0
I20260811 22:20:51.868687 11045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab.
Entry for principal oryx/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab.
I20260811 22:20:51.905366 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42757
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:52.019363 11063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:52.019649 11063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:52.019721 11063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:52.023803 11063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:52.023860 11063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:52.023914 11063 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:52.023970 11063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:52.029276 11063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:52.030390 11063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:52.031420 11063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:52.040771 11072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:52.040808 11076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:52.040781 11074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:52.041024 11063 server_base.cc:1034] running on GCE node
I20260811 22:20:52.041489 11063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:52.042160 11063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:52.043323 11063 hybrid_clock.cc:648] HybridClock initialized: now 1786486852043295 us; error 44 us; skew 500 ppm
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:52.046149 11063 init.cc:377] Logged in from keytab as oryx/127.29.3.65@KRBTEST.COM (short username oryx)
I20260811 22:20:52.047003 11063 webserver.cc:460] Webserver started at http://127.29.3.65:42797/ using document root <none> and password file <none>
I20260811 22:20:52.047534 11063 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:52.047595 11063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:52.047798 11063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:52.049835 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "359b52db26994a78994e5ae8f2cb26a8"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "7371771fd8f7e4ea7d0448737dd0ed85"
server_key_iv: "5bd680ccb1aa0eee2312bb75f164682f"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.050408 11063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "359b52db26994a78994e5ae8f2cb26a8"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "7371771fd8f7e4ea7d0448737dd0ed85"
server_key_iv: "5bd680ccb1aa0eee2312bb75f164682f"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.053977 11063 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:52.056398 11086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:52.057616 11063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:20:52.057717 11063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "359b52db26994a78994e5ae8f2cb26a8"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "7371771fd8f7e4ea7d0448737dd0ed85"
server_key_iv: "5bd680ccb1aa0eee2312bb75f164682f"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.057819 11063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:52.094192 11063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:52.097045 11063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:52.097234 11063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:52.098784 11063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:52.098840 11063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:52.098896 11063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:52.098927 11063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:52.109313 11063 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:42097
I20260811 22:20:52.109385 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:42097 every 8 connection(s)
I20260811 22:20:52.110349 11063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:52.112263 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.099488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:35653 (local address 127.29.3.126:42757)
0811 22:20:52.099614 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.099618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.100705 (+ 1087us) server_negotiation.cc:407] Connection header received
0811 22:20:52.101697 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.101701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.101764 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.101824 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.103454 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.104282 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.105538 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.105779 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.105875 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.108544 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.108547 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.108549 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.108579 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.110105 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.110700 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.110704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.110705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.110814 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.111227 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.111231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.111233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.111476 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.111562 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.111998 (+ 436us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.112111 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:20:52.112504 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 11063
I20260811 22:20:52.112625 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:52.112957 29709 external_mini_cluster.cc:1428] Setting key 595b5d35f2ddcec0572e625957fac7af
I20260811 22:20:52.113204 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.100035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.29.3.65:35653)
0811 22:20:52.100549 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.100594 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.101432 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.102254 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.102264 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.102722 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.103282 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.103302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.104422 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.104426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.105417 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.105430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.105942 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.105947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.106117 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.106716 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.106733 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.108367 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.110221 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.110227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.110241 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.110597 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.110928 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.110931 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.110933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.111103 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.111565 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.111570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.111799 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.111949 (+ 150us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.112211 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":141,"threads_started":1}
I20260811 22:20:52.114511 11281 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42757
I20260811 22:20:52.114687 11281 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:52.115268 11281 heartbeater.cc:511] Master 127.29.3.126:42757 requested a full tablet report, sending...
I20260811 22:20:52.116915 10947 ts_manager.cc:194] Registered new tserver with Master: 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097)
I20260811 22:20:52.118037 10947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:35653
I20260811 22:20:52.123904 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.116620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.29.3.65:50749)
0811 22:20:52.116893 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.116908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.117008 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.117346 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.117348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.117544 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.117663 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.117669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.118645 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.118647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.119390 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.119402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.119946 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.119950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.120043 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.120497 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.120508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.120822 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.122537 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.122539 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.122541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.122765 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.123021 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.123023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.123025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.123093 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.123469 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.123471 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.123541 (+ 70us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.123612 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.123714 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
I20260811 22:20:52.124006 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.116698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:50749 (local address 127.29.3.126:42757)
0811 22:20:52.116823 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.116827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.116980 (+ 153us) server_negotiation.cc:407] Connection header received
0811 22:20:52.117119 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.117122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.117166 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.117213 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.117802 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.118521 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.119523 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.119821 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.119936 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.120958 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.120962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.120965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.120990 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.122439 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.122849 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.122853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.122854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.122941 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.123166 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.123169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.123171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.123386 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.123490 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.123668 (+ 178us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.123859 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for oryx/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab.
Entry for principal oryx/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab.
I20260811 22:20:52.144865 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42757
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:52.258024 11293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:52.258335 11293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:52.258392 11293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:52.262465 11293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:52.262543 11293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:52.262601 11293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:52.262658 11293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:52.268029 11293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:52.269140 11293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:52.270172 11293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:52.279938 11302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:52.279983 11293 server_base.cc:1034] running on GCE node
W20260811 22:20:52.279917 11307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:52.279942 11304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:52.280558 11293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:52.281205 11293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:52.282353 11293 hybrid_clock.cc:648] HybridClock initialized: now 1786486852282336 us; error 28 us; skew 500 ppm
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:52.284917 11293 init.cc:377] Logged in from keytab as oryx/127.29.3.66@KRBTEST.COM (short username oryx)
I20260811 22:20:52.285763 11293 webserver.cc:460] Webserver started at http://127.29.3.66:41163/ using document root <none> and password file <none>
I20260811 22:20:52.286317 11293 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:52.286391 11293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:52.286597 11293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:52.288597 11293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "50e7334048de452db11ecf7cd7610561"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "7b801c69952a7944d6d747b94f34c760"
server_key_iv: "eaaac11194e23b66d50f7b35d78963b0"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.289144 11293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "50e7334048de452db11ecf7cd7610561"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "7b801c69952a7944d6d747b94f34c760"
server_key_iv: "eaaac11194e23b66d50f7b35d78963b0"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.292829 11293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260811 22:20:52.295254 11317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:52.296458 11293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:52.296566 11293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "50e7334048de452db11ecf7cd7610561"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "7b801c69952a7944d6d747b94f34c760"
server_key_iv: "eaaac11194e23b66d50f7b35d78963b0"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.296660 11293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:52.316910 11293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:52.320163 11293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:52.320336 11293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:52.321906 11293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:52.321962 11293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:52.322018 11293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:52.322047 11293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:52.331892 11293 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:46245
I20260811 22:20:52.331950 11517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:46245 every 8 connection(s)
I20260811 22:20:52.332952 11293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:52.335634 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.322704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:37569 (local address 127.29.3.126:42757)
0811 22:20:52.322870 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.322876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.323773 (+ 897us) server_negotiation.cc:407] Connection header received
0811 22:20:52.324894 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.324898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.324962 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.325022 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.326831 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.327418 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.329084 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.329319 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.329408 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.332372 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.332375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.332377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.332406 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.333895 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.334379 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.334382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.334383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.334459 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.334728 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.334730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.334731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.334899 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.334969 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.335409 (+ 440us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.335510 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:20:52.336423 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.323104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.29.3.66:37569)
0811 22:20:52.323601 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.323652 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.324623 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.325514 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.325524 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.325939 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.326674 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.326694 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.327547 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.327550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.328950 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.328962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.329467 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.329471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.329647 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.330379 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.330400 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.332188 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.333994 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.333999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.334013 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.334271 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.334525 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.334527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.334529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.334656 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.334969 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.334974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.335216 (+ 242us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.335367 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.335613 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":103,"threads_started":1}
I20260811 22:20:52.337546 11518 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42757
I20260811 22:20:52.337692 11518 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:52.338333 11518 heartbeater.cc:511] Master 127.29.3.126:42757 requested a full tablet report, sending...
I20260811 22:20:52.339454 10945 ts_manager.cc:194] Registered new tserver with Master: 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245)
I20260811 22:20:52.340058 10945 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:37569
I20260811 22:20:52.342689 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 11293
I20260811 22:20:52.342794 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:52.343075 29709 external_mini_cluster.cc:1428] Setting key 51aa3643bf00536efcfd6d93651eed4a
I20260811 22:20:52.346910 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.339958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.29.3.66:50343)
0811 22:20:52.340397 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.340412 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.340504 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.340736 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.340739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.340938 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.341032 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.341037 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.341769 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.341773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.342358 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.342368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.342780 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.342783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.342883 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.343377 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.343388 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.343768 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.345463 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.345466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.345467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.345648 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.345900 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.345902 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.345903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.345970 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.346412 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.346414 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.346491 (+ 77us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.346569 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.346708 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"spinlock_wait_cycles":224640,"thread_start_us":105,"threads_started":1}
I20260811 22:20:52.346916 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.340045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:50343 (local address 127.29.3.126:42757)
0811 22:20:52.340282 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.340286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.340464 (+ 178us) server_negotiation.cc:407] Connection header received
0811 22:20:52.340597 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.340599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.340638 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.340699 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.341140 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.341639 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.342472 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.342679 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.342764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.343873 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.343876 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.343878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.343901 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.345349 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.345730 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.345733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.345735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.345821 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.346035 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.346037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.346039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.346330 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.346429 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.346630 (+ 201us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.346765 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab.
Entry for principal oryx/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab.
I20260811 22:20:52.371297 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:42757
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:52.483685 11534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:52.483922 11534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:52.483965 11534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:52.487867 11534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:52.487920 11534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:52.487957 11534 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 22:20:52.487993 11534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:52.493049 11534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.3.126:42757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:52.494030 11534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:52.495018 11534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:52.504405 11543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:52.504405 11541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:52.504482 11534 server_base.cc:1034] running on GCE node
W20260811 22:20:52.504415 11546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:52.505039 11534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:52.505679 11534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:52.506825 11534 hybrid_clock.cc:648] HybridClock initialized: now 1786486852506799 us; error 39 us; skew 500 ppm
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:52.509433 11534 init.cc:377] Logged in from keytab as oryx/127.29.3.67@KRBTEST.COM (short username oryx)
I20260811 22:20:52.510260 11534 webserver.cc:460] Webserver started at http://127.29.3.67:40099/ using document root <none> and password file <none>
I20260811 22:20:52.510811 11534 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:52.510876 11534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:52.511062 11534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:52.513044 11534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "3680fa72e4f3406cb973e80ef4fb8783"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "1327625b4f0293068c83d727e554f789"
server_key_iv: "ede16d31fe852f6f84e381e3524ef61d"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.513583 11534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "3680fa72e4f3406cb973e80ef4fb8783"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "1327625b4f0293068c83d727e554f789"
server_key_iv: "ede16d31fe852f6f84e381e3524ef61d"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.517306 11534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260811 22:20:52.519731 11557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:52.520870 11534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:52.520980 11534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "3680fa72e4f3406cb973e80ef4fb8783"
format_stamp: "Formatted at 2026-08-11 22:20:52 on dist-test-slave-4tdj"
server_key: "1327625b4f0293068c83d727e554f789"
server_key_iv: "ede16d31fe852f6f84e381e3524ef61d"
server_key_version: "encryptionkey@0"
I20260811 22:20:52.521073 11534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:52.551754 11534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:52.555017 11534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:52.555224 11534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:52.556829 11534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:52.556895 11534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:52.556951 11534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:52.556977 11534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:52.567443 11534 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:42065
I20260811 22:20:52.567515 11764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:42065 every 8 connection(s)
I20260811 22:20:52.568452 11534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:52.570770 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.557563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:40775 (local address 127.29.3.126:42757)
0811 22:20:52.557676 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.557680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.558851 (+ 1171us) server_negotiation.cc:407] Connection header received
0811 22:20:52.559808 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.559812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.559873 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.559929 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.561637 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.562316 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.564009 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.564357 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.564468 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.567360 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.567363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.567366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.567397 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.568853 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.569427 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.569430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.569431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.569513 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.569795 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.569796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.569798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.569968 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.570043 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.570525 (+ 482us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.570628 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:20:52.571558 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.558165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.29.3.67:40775)
0811 22:20:52.558690 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.558745 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.559547 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.560362 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.560371 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.560839 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.561485 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.561502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.562447 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.562451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.563881 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.563894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.564537 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.564542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.564748 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.565357 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.565380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.567196 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.568985 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.568991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.569005 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.569329 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.569582 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.569584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.569585 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.569727 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.570041 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.570045 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.570276 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.570473 (+ 197us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.570725 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":136,"threads_started":1}
I20260811 22:20:52.572711 11765 heartbeater.cc:347] Connected to a master server at 127.29.3.126:42757
I20260811 22:20:52.572856 11765 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:52.573421 11765 heartbeater.cc:511] Master 127.29.3.126:42757 requested a full tablet report, sending...
I20260811 22:20:52.574517 10947 ts_manager.cc:194] Registered new tserver with Master: 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065)
I20260811 22:20:52.575227 10947 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:40775
I20260811 22:20:52.578670 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 11534
I20260811 22:20:52.578819 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:52.579142 29709 external_mini_cluster.cc:1428] Setting key 390d48716528b92ca6a9fd0dcf7edda3
I20260811 22:20:52.581329 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:20:52.581579 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.575014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:46825 (local address 127.29.3.126:42757)
0811 22:20:52.575118 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.575122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.575291 (+ 169us) server_negotiation.cc:407] Connection header received
0811 22:20:52.575397 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.575400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.575443 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.575506 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.575956 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.576443 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.577403 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.577596 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.577662 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.578674 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.578678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.578681 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.578707 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.580073 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.580437 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.580440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.580442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.580525 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.580766 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.580769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.580771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.580973 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.581069 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.581332 (+ 263us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.581436 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 22:20:52.581593 11772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.574955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.29.3.67:46825)
0811 22:20:52.575218 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.575229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.575320 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.575531 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.575534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.575710 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.575838 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.575846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.576565 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.576568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.577315 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.577325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.577672 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.577674 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.577738 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.578194 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.578205 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.578550 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.580169 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.580172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.580173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.580353 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.580610 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.580612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.580613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.580689 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.581055 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.581058 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.581155 (+ 97us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.581268 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.581405 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:20:52.610809 11027 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:52.600060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.29.3.126:42757)
0811 22:20:52.600181 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.600186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.600202 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:20:52.600251 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.600253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.600297 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.600397 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.601352 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.602144 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.603414 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.603766 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.603877 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.610621 (+ 6744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46836: BlockingRecv error: recv got EOF from 127.0.0.1:46836 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486852, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:52.620919 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.613212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.29.3.126:42757)
0811 22:20:52.613321 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.613325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.613521 (+ 196us) server_negotiation.cc:407] Connection header received
0811 22:20:52.613681 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.613685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.613739 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.613789 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.614531 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.615043 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.615826 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.616040 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.616120 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.618203 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.618206 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.618208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.618236 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.619599 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.619984 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.619986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.619987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.620061 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.620261 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.620263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.620264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.620433 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.620535 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.620689 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.620794 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 22:20:52.623775 10947 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:52.626924 10947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:52.640969 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.635129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:46245 (local address 127.0.0.1:39504)
0811 22:20:52.635863 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.635876 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.635969 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.637036 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.637038 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.637054 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:52.637120 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.637125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.638439 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.638442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.639561 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.639571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.640539 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.640543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.640656 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.640682 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.640734 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.640808 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260811 22:20:52.641055 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.635100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.29.3.67:42065)
0811 22:20:52.635536 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.635547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.635574 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:20:52.635850 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.635856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.636066 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.636221 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:52.636713 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.637839 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.639519 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.640327 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.640434 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.640595 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.640805 (+ 210us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.640920 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":123,"threads_started":1}
I20260811 22:20:52.640838 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.635697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:42097 (local address 127.0.0.1:40176)
0811 22:20:52.636272 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.636284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.636364 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.636913 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.636916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.636933 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:52.637080 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.637087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.638407 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.638410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.639497 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.639509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.640298 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.640301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.640417 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.640455 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.640561 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.640682 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":52,"thread_start_us":105,"threads_started":1}
I20260811 22:20:52.641286 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.636103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.29.3.65:42097)
0811 22:20:52.636482 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.636488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.636510 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:20:52.636581 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.636586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.636779 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.636922 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:52.637199 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.638261 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.639621 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.640175 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.640278 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.640439 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.641053 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.641157 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":100,"threads_started":1}
I20260811 22:20:52.640972 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.634974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:42065 (local address 127.0.0.1:38078)
0811 22:20:52.635444 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.635494 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.635619 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.636399 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.636405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.636431 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:52.636573 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.636581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.637966 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.637970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.639392 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.639404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.640515 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.640519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.640629 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.640650 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.640723 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.640781 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260811 22:20:52.641757 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.635768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.29.3.66:46245)
0811 22:20:52.636611 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.636623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.636646 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:20:52.636732 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.636737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.636892 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.637254 (+ 362us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:52.637400 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.638278 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.639659 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.640411 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.640924 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.641433 (+ 509us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.641539 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.641624 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":92,"threads_started":1}
I20260811 22:20:52.644251 11405 tablet_service.cc:1467] Processing CreateTablet for tablet c9a42ff101254d2c8b0e46d49ca1daa8 (DEFAULT_TABLE table=test-table [id=22fcd24a92514b29a4fe26cda4d0c3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:52.644285 11645 tablet_service.cc:1467] Processing CreateTablet for tablet c9a42ff101254d2c8b0e46d49ca1daa8 (DEFAULT_TABLE table=test-table [id=22fcd24a92514b29a4fe26cda4d0c3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:52.644272 11168 tablet_service.cc:1467] Processing CreateTablet for tablet c9a42ff101254d2c8b0e46d49ca1daa8 (DEFAULT_TABLE table=test-table [id=22fcd24a92514b29a4fe26cda4d0c3ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:52.645275 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a42ff101254d2c8b0e46d49ca1daa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.645321 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a42ff101254d2c8b0e46d49ca1daa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.645321 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a42ff101254d2c8b0e46d49ca1daa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.652140 11816 tablet_bootstrap.cc:492] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Bootstrap starting.
I20260811 22:20:52.653138 11820 tablet_bootstrap.cc:492] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Bootstrap starting.
I20260811 22:20:52.653594 11819 tablet_bootstrap.cc:492] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Bootstrap starting.
I20260811 22:20:52.654436 11816 tablet_bootstrap.cc:654] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.655154 11816 log.cc:824] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:52.655303 11820 tablet_bootstrap.cc:654] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.655577 11819 tablet_bootstrap.cc:654] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.656144 11820 log.cc:824] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:52.656319 11819 log.cc:824] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:52.657282 11816 tablet_bootstrap.cc:492] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: No bootstrap required, opened a new log
I20260811 22:20:52.657469 11816 ts_tablet_manager.cc:1397] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260811 22:20:52.658072 11820 tablet_bootstrap.cc:492] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: No bootstrap required, opened a new log
I20260811 22:20:52.658072 11819 tablet_bootstrap.cc:492] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: No bootstrap required, opened a new log
I20260811 22:20:52.658262 11820 ts_tablet_manager.cc:1397] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 22:20:52.658262 11819 ts_tablet_manager.cc:1397] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260811 22:20:52.660377 11816 raft_consensus.cc:348] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.660594 11816 raft_consensus.cc:374] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.660646 11816 raft_consensus.cc:729] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7334048de452db11ecf7cd7610561, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.661059 11816 consensus_queue.cc:260] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.661784 11518 heartbeater.cc:503] Master 127.29.3.126:42757 was elected leader, sending a full tablet report...
I20260811 22:20:52.662072 11816 ts_tablet_manager.cc:1428] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260811 22:20:52.662730 11819 raft_consensus.cc:348] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.662961 11819 raft_consensus.cc:374] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.662807 11820 raft_consensus.cc:348] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.663015 11819 raft_consensus.cc:729] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.663049 11820 raft_consensus.cc:374] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.663112 11820 raft_consensus.cc:729] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.663487 11819 consensus_queue.cc:260] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.663516 11820 consensus_queue.cc:260] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.664424 11819 ts_tablet_manager.cc:1428] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:20:52.664460 11820 ts_tablet_manager.cc:1428] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:20:52.664680 11765 heartbeater.cc:503] Master 127.29.3.126:42757 was elected leader, sending a full tablet report...
I20260811 22:20:52.664703 11281 heartbeater.cc:503] Master 127.29.3.126:42757 was elected leader, sending a full tablet report...
I20260811 22:20:52.720194 11828 raft_consensus.cc:482] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:52.720405 11828 raft_consensus.cc:504] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.721520 11828 leader_election.cc:288] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245), 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097)
I20260811 22:20:52.725922 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.721917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:52249 (local address 127.29.3.66:46245)
0811 22:20:52.722055 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.722059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.722074 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:52.722202 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.722206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.722259 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.722373 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:52.722675 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.723456 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.724765 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.725384 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.725486 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.725545 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.725737 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.725795 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:20:52.725925 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.721800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:46245 (local address 127.29.3.67:52249)
0811 22:20:52.721916 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.721932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.722056 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.722392 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.722395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.722419 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:52.722545 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.722551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.723554 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.723557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.724621 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.724632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.725502 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.725506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.725611 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.725638 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.725721 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.725792 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 22:20:52.726428 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.721800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:42097 (local address 127.29.3.67:43663)
0811 22:20:52.722328 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.722342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.722437 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.722670 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.722673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.722689 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:52.723266 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.723277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.724216 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.724219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.725147 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.725158 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.725985 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.725988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.726088 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.726116 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.726193 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.726260 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":126,"thread_start_us":128,"threads_started":1}
I20260811 22:20:52.726432 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.721917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43663 (local address 127.29.3.65:42097)
0811 22:20:52.722038 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.722043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.722428 (+ 385us) server_negotiation.cc:407] Connection header received
0811 22:20:52.722522 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.722525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.722576 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.722646 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:52.723378 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.724106 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.725274 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.725866 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.725961 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.726022 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.726219 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.726320 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:52.726735 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a42ff101254d2c8b0e46d49ca1daa8" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561" is_pre_election: true
I20260811 22:20:52.727033 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a42ff101254d2c8b0e46d49ca1daa8" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359b52db26994a78994e5ae8f2cb26a8" is_pre_election: true
I20260811 22:20:52.727082 11441 raft_consensus.cc:2454] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 0.
I20260811 22:20:52.727336 11209 raft_consensus.cc:2454] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 0.
I20260811 22:20:52.727604 11559 leader_election.cc:302] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3680fa72e4f3406cb973e80ef4fb8783, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:52.727933 11828 raft_consensus.cc:2781] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:52.728013 11828 raft_consensus.cc:482] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:52.728044 11828 raft_consensus.cc:3037] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:52.729120 11828 raft_consensus.cc:504] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.729455 11828 leader_election.cc:288] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 election: Requested vote from peers 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245), 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097)
I20260811 22:20:52.729806 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a42ff101254d2c8b0e46d49ca1daa8" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359b52db26994a78994e5ae8f2cb26a8"
I20260811 22:20:52.729912 11209 raft_consensus.cc:3037] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:52.731029 11209 raft_consensus.cc:2454] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 1.
I20260811 22:20:52.731293 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9a42ff101254d2c8b0e46d49ca1daa8" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561"
I20260811 22:20:52.731329 11562 leader_election.cc:302] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 3680fa72e4f3406cb973e80ef4fb8783; no voters:
I20260811 22:20:52.731408 11441 raft_consensus.cc:3037] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:52.731478 11828 raft_consensus.cc:2781] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:52.731716 11828 raft_consensus.cc:686] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 LEADER]: Becoming Leader. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Running, Role: LEADER
I20260811 22:20:52.732029 11828 consensus_queue.cc:237] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } }
I20260811 22:20:52.732818 11441 raft_consensus.cc:2454] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 1.
I20260811 22:20:52.735049 10945 catalog_manager.cc:5495] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "3680fa72e4f3406cb973e80ef4fb8783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:52.762583 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.759372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38094 (local address 127.29.3.67:42065)
0811 22:20:52.759489 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.759492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.759526 (+ 34us) server_negotiation.cc:407] Connection header received
0811 22:20:52.759603 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.759605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.759664 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.759756 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:52.759935 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.760425 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.761209 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.761450 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.761525 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.761784 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:52.762233 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:52.762370 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.762420 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.762473 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:20:52.768390 11209 raft_consensus.cc:1261] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Refusing update from remote peer 3680fa72e4f3406cb973e80ef4fb8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:52.768389 11441 raft_consensus.cc:1261] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Refusing update from remote peer 3680fa72e4f3406cb973e80ef4fb8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:52.769083 11828 consensus_queue.cc:1035] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:52.769271 11831 consensus_queue.cc:1035] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:52.776307 11835 mvcc.cc:205] Tried to move back new op lower bound from 7317450148932259840 to 7317450148793249792. Current Snapshot: MvccSnapshot[applied={T|T < 7317450148932259840}]
I20260811 22:20:52.776566 11840 mvcc.cc:205] Tried to move back new op lower bound from 7317450148932259840 to 7317450148793249792. Current Snapshot: MvccSnapshot[applied={T|T < 7317450148932259840}]
I20260811 22:20:52.778052 11837 mvcc.cc:205] Tried to move back new op lower bound from 7317450148932259840 to 7317450148793249792. Current Snapshot: MvccSnapshot[applied={T|T < 7317450148932259840}]
I20260811 22:20:52.785136 10946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:52.785360 10946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:52.788000 10946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Sending DeleteTablet for 3 replicas of tablet c9a42ff101254d2c8b0e46d49ca1daa8
I20260811 22:20:52.788865 11645 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a42ff101254d2c8b0e46d49ca1daa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:52 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38078
I20260811 22:20:52.788965 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a42ff101254d2c8b0e46d49ca1daa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:52 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39504
I20260811 22:20:52.789012 11168 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a42ff101254d2c8b0e46d49ca1daa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:52 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40176
I20260811 22:20:52.789868 11854 tablet_replica.cc:330] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: stopping tablet replica
I20260811 22:20:52.790082 10946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:52.790508 10946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:52.791494 11854 raft_consensus.cc:2229] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:52.791864 11854 raft_consensus.cc:2258] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:52.792284 11855 tablet_replica.cc:330] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: stopping tablet replica
I20260811 22:20:52.792922 11855 raft_consensus.cc:2229] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:52.794785 11854 ts_tablet_manager.cc:1905] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:52.795400 11405 tablet_service.cc:1467] Processing CreateTablet for tablet 43b0d878da874f8a9ec904b4f3137751 (DEFAULT_TABLE table=test-table [id=806a3a08f55e4a60a1d68622cd1846b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:52.795657 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b0d878da874f8a9ec904b4f3137751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.795693 11168 tablet_service.cc:1467] Processing CreateTablet for tablet 43b0d878da874f8a9ec904b4f3137751 (DEFAULT_TABLE table=test-table [id=806a3a08f55e4a60a1d68622cd1846b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:52.795970 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b0d878da874f8a9ec904b4f3137751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.797658 11854 ts_tablet_manager.cc:1918] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:52.797744 11854 log.cc:1196] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/c9a42ff101254d2c8b0e46d49ca1daa8
I20260811 22:20:52.797923 11816 tablet_bootstrap.cc:492] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Bootstrap starting.
I20260811 22:20:52.798054 11854 ts_tablet_manager.cc:1939] T c9a42ff101254d2c8b0e46d49ca1daa8 P 50e7334048de452db11ecf7cd7610561: Deleting consensus metadata
I20260811 22:20:52.798877 11816 tablet_bootstrap.cc:654] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.799391 11855 raft_consensus.cc:2258] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:52.799463 11819 tablet_bootstrap.cc:492] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Bootstrap starting.
I20260811 22:20:52.799888 11859 tablet_replica.cc:330] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: stopping tablet replica
I20260811 22:20:52.800439 11819 tablet_bootstrap.cc:654] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.801008 11855 ts_tablet_manager.cc:1905] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:52.802603 11859 raft_consensus.cc:2229] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:52.802898 11859 raft_consensus.cc:2258] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:52.803637 11855 ts_tablet_manager.cc:1918] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:52.803717 11855 log.cc:1196] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/c9a42ff101254d2c8b0e46d49ca1daa8
I20260811 22:20:52.803987 11855 ts_tablet_manager.cc:1939] T c9a42ff101254d2c8b0e46d49ca1daa8 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting consensus metadata
I20260811 22:20:52.805533 11645 tablet_service.cc:1467] Processing CreateTablet for tablet 43b0d878da874f8a9ec904b4f3137751 (DEFAULT_TABLE table=test-table [id=806a3a08f55e4a60a1d68622cd1846b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:52.805780 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b0d878da874f8a9ec904b4f3137751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.805820 10922 catalog_manager.cc:4841] TS 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065): tablet c9a42ff101254d2c8b0e46d49ca1daa8 (table test-table [id=22fcd24a92514b29a4fe26cda4d0c3ca]) successfully deleted
I20260811 22:20:52.807350 11859 ts_tablet_manager.cc:1905] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:52.808332 10918 catalog_manager.cc:4841] TS 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245): tablet c9a42ff101254d2c8b0e46d49ca1daa8 (table test-table [id=22fcd24a92514b29a4fe26cda4d0c3ca]) successfully deleted
I20260811 22:20:52.809232 11816 tablet_bootstrap.cc:492] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: No bootstrap required, opened a new log
I20260811 22:20:52.809373 11816 ts_tablet_manager.cc:1397] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.002s
I20260811 22:20:52.809881 11816 raft_consensus.cc:348] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.809994 11816 raft_consensus.cc:374] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.810022 11816 raft_consensus.cc:729] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7334048de452db11ecf7cd7610561, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.810140 11816 consensus_queue.cc:260] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.810755 11816 ts_tablet_manager.cc:1428] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:52.811053 11820 tablet_bootstrap.cc:492] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Bootstrap starting.
I20260811 22:20:52.810735 11859 ts_tablet_manager.cc:1918] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:52.811699 11859 log.cc:1196] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/c9a42ff101254d2c8b0e46d49ca1daa8
I20260811 22:20:52.812074 11820 tablet_bootstrap.cc:654] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.812485 11859 ts_tablet_manager.cc:1939] T c9a42ff101254d2c8b0e46d49ca1daa8 P 359b52db26994a78994e5ae8f2cb26a8: Deleting consensus metadata
I20260811 22:20:52.813772 11819 tablet_bootstrap.cc:492] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: No bootstrap required, opened a new log
I20260811 22:20:52.813827 10922 catalog_manager.cc:4841] TS 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097): tablet c9a42ff101254d2c8b0e46d49ca1daa8 (table test-table [id=22fcd24a92514b29a4fe26cda4d0c3ca]) successfully deleted
I20260811 22:20:52.813862 11819 ts_tablet_manager.cc:1397] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20260811 22:20:52.814330 11819 raft_consensus.cc:348] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.814440 11819 raft_consensus.cc:374] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.814473 11819 raft_consensus.cc:729] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.814589 11819 consensus_queue.cc:260] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.814868 11819 ts_tablet_manager.cc:1428] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:52.815948 11820 tablet_bootstrap.cc:492] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: No bootstrap required, opened a new log
I20260811 22:20:52.816032 11820 ts_tablet_manager.cc:1397] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 22:20:52.816494 11820 raft_consensus.cc:348] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.816599 11820 raft_consensus.cc:374] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.816632 11820 raft_consensus.cc:729] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.816753 11820 consensus_queue.cc:260] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.817019 11820 ts_tablet_manager.cc:1428] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260811 22:20:52.819341 11767 tablet.cc:2369] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:20:52.834195 11519 tablet.cc:2369] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:52.859704 11845 raft_consensus.cc:482] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:52.859853 11845 raft_consensus.cc:504] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.860323 11845 leader_election.cc:288] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097), 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245)
I20260811 22:20:52.860747 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b0d878da874f8a9ec904b4f3137751" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359b52db26994a78994e5ae8f2cb26a8" is_pre_election: true
I20260811 22:20:52.860903 11209 raft_consensus.cc:2454] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 0.
I20260811 22:20:52.860827 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b0d878da874f8a9ec904b4f3137751" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561" is_pre_election: true
I20260811 22:20:52.861006 11441 raft_consensus.cc:2454] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 0.
I20260811 22:20:52.861204 11562 leader_election.cc:302] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 3680fa72e4f3406cb973e80ef4fb8783; no voters:
I20260811 22:20:52.861397 11845 raft_consensus.cc:2781] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:52.861459 11845 raft_consensus.cc:482] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:52.861490 11845 raft_consensus.cc:3037] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Advancing to term 1
W20260811 22:20:52.861747 11284 tablet.cc:2369] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:20:52.862690 11845 raft_consensus.cc:504] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.863056 11845 leader_election.cc:288] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 election: Requested vote from peers 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097), 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245)
I20260811 22:20:52.863435 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b0d878da874f8a9ec904b4f3137751" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561"
I20260811 22:20:52.863437 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b0d878da874f8a9ec904b4f3137751" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359b52db26994a78994e5ae8f2cb26a8"
I20260811 22:20:52.863559 11209 raft_consensus.cc:3037] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:52.863562 11441 raft_consensus.cc:3037] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:52.864499 11441 raft_consensus.cc:2454] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 1.
I20260811 22:20:52.864710 11209 raft_consensus.cc:2454] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 1.
I20260811 22:20:52.864782 11559 leader_election.cc:302] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3680fa72e4f3406cb973e80ef4fb8783, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:52.864959 11845 raft_consensus.cc:2781] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:52.865051 11845 raft_consensus.cc:686] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 LEADER]: Becoming Leader. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Running, Role: LEADER
I20260811 22:20:52.865197 11845 consensus_queue.cc:237] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.867031 10946 catalog_manager.cc:5495] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "3680fa72e4f3406cb973e80ef4fb8783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } health_report { overall_health: HEALTHY } } }
Aug 11 22:20:52 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486851, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.126@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:52.925515 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.916292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46860 (local address 127.29.3.126:42757)
0811 22:20:52.916432 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:52.916437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:52.916646 (+ 209us) server_negotiation.cc:407] Connection header received
0811 22:20:52.916797 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:52.916799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:52.916840 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:52.916909 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:52.917860 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.918407 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.919471 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.919686 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.919755 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.922166 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:52.922169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:52.922171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:52.922193 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:52.923597 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.924129 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.924148 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.924149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.924222 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:52.924526 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:52.924528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:52.924529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:52.924749 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:52.924839 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:52.925135 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 22:20:52.925327 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":55}
I20260811 22:20:52.925690 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:52.916211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:42757 (local address 127.0.0.1:46860)
0811 22:20:52.916538 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:52.916555 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:52.916706 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:52.916965 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:52.916968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:52.917589 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:52.917720 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.917729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.918517 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.918521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.919366 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:52.919378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:52.919777 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:52.919781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:52.919878 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:52.920433 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:52.920447 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:52.922035 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:52.923729 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.923738 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.923745 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.924045 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.924327 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:52.924330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:52.924332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:52.924450 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:52.924849 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:52.924857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:52.924971 (+ 114us) client_negotiation.cc:769] Sending connection context
0811 22:20:52.925072 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 22:20:52.925381 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":80,"thread_start_us":95,"threads_started":1}
I20260811 22:20:52.929718 10946 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:46860:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 22:20:52.930534 10946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:52.935561 11168 tablet_service.cc:1467] Processing CreateTablet for tablet f3dd0722cce145b9b051f586d01fc61b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24486f3f62d54e489c0684b8301888dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:52.935715 11645 tablet_service.cc:1467] Processing CreateTablet for tablet f3dd0722cce145b9b051f586d01fc61b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24486f3f62d54e489c0684b8301888dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:52.935964 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd0722cce145b9b051f586d01fc61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.936072 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd0722cce145b9b051f586d01fc61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.938529 11819 tablet_bootstrap.cc:492] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: Bootstrap starting.
I20260811 22:20:52.938961 11405 tablet_service.cc:1467] Processing CreateTablet for tablet f3dd0722cce145b9b051f586d01fc61b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24486f3f62d54e489c0684b8301888dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 22:20:52.939256 11820 tablet_bootstrap.cc:492] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: Bootstrap starting.
I20260811 22:20:52.939321 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3dd0722cce145b9b051f586d01fc61b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:52.939543 11819 tablet_bootstrap.cc:654] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.940056 11820 tablet_bootstrap.cc:654] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.941174 11820 tablet_bootstrap.cc:492] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: No bootstrap required, opened a new log
I20260811 22:20:52.941253 11820 ts_tablet_manager.cc:1397] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:52.941680 11820 raft_consensus.cc:348] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.941798 11820 raft_consensus.cc:374] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.941828 11820 raft_consensus.cc:729] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.942004 11816 tablet_bootstrap.cc:492] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: Bootstrap starting.
I20260811 22:20:52.942008 11820 consensus_queue.cc:260] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.942334 11820 ts_tablet_manager.cc:1428] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260811 22:20:52.942535 11845 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:52.942665 11845 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:52.942929 11816 tablet_bootstrap.cc:654] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:52.942977 11819 tablet_bootstrap.cc:492] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: No bootstrap required, opened a new log
I20260811 22:20:52.943053 11819 ts_tablet_manager.cc:1397] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 22:20:52.943495 11819 raft_consensus.cc:348] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.943605 11819 raft_consensus.cc:374] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.943651 11819 raft_consensus.cc:729] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.943768 11819 consensus_queue.cc:260] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.944054 11819 ts_tablet_manager.cc:1428] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:52.944307 11827 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:52.944450 11827 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:52.944792 11816 tablet_bootstrap.cc:492] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: No bootstrap required, opened a new log
I20260811 22:20:52.944867 11816 ts_tablet_manager.cc:1397] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:52.945274 11816 raft_consensus.cc:348] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.945355 11816 raft_consensus.cc:374] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:52.945374 11816 raft_consensus.cc:729] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7334048de452db11ecf7cd7610561, State: Initialized, Role: FOLLOWER
I20260811 22:20:52.945473 11816 consensus_queue.cc:260] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:52.945710 11826 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:52.945844 11826 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:52.945710 11816 ts_tablet_manager.cc:1428] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:53.142647 11827 raft_consensus.cc:482] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:53.142814 11827 raft_consensus.cc:504] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.143987 11827 leader_election.cc:288] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245), 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065)
I20260811 22:20:53.147989 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.144370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:58613 (local address 127.29.3.67:42065)
0811 22:20:53.144519 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.144523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.144888 (+ 365us) server_negotiation.cc:407] Connection header received
0811 22:20:53.144996 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.144999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.145048 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.145121 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:53.145372 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.145852 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.147007 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.147411 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.147505 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.147572 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.147790 (+ 218us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.147862 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:53.147990 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.144313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:35813 (local address 127.29.3.66:46245)
0811 22:20:53.144463 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.144468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.144724 (+ 256us) server_negotiation.cc:407] Connection header received
0811 22:20:53.144866 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.144870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.144920 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.144994 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:20:53.145276 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.145809 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.147018 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.147406 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.147498 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.147569 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.147778 (+ 209us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.147854 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:20:53.148008 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.144264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:46245 (local address 127.29.3.65:35813)
0811 22:20:53.144622 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.144637 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:53.144760 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:53.145022 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:53.145025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:53.145041 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:53.145156 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.145163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.145927 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.145931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.146892 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.146903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.147520 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.147523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.147656 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.147676 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 22:20:53.147747 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 22:20:53.147844 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":73,"thread_start_us":72,"threads_started":1}
I20260811 22:20:53.148011 11887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.144272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:42065 (local address 127.29.3.65:58613)
0811 22:20:53.144788 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.144800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:53.144908 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:53.145135 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:53.145138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:53.145151 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:20:53.145258 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.145265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.145973 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.145976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.146884 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.146895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.147546 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.147549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.147675 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.147694 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:20:53.147762 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 22:20:53.147870 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":237,"thread_start_us":114,"threads_started":1}
I20260811 22:20:53.148620 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd0722cce145b9b051f586d01fc61b" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561" is_pre_election: true
I20260811 22:20:53.148774 11441 raft_consensus.cc:2454] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 0.
I20260811 22:20:53.148768 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd0722cce145b9b051f586d01fc61b" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3680fa72e4f3406cb973e80ef4fb8783" is_pre_election: true
I20260811 22:20:53.149024 11679 raft_consensus.cc:2454] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 0.
I20260811 22:20:53.149139 11088 leader_election.cc:302] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:53.149420 11827 raft_consensus.cc:2781] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:53.149489 11827 raft_consensus.cc:482] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:53.149536 11827 raft_consensus.cc:3037] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.150878 11827 raft_consensus.cc:504] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.151201 11827 leader_election.cc:288] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 election: Requested vote from peers 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245), 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065)
I20260811 22:20:53.151543 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd0722cce145b9b051f586d01fc61b" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3680fa72e4f3406cb973e80ef4fb8783"
I20260811 22:20:53.151539 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3dd0722cce145b9b051f586d01fc61b" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561"
I20260811 22:20:53.151652 11679 raft_consensus.cc:3037] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.151659 11441 raft_consensus.cc:3037] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.152558 11679 raft_consensus.cc:2454] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 1.
I20260811 22:20:53.152572 11441 raft_consensus.cc:2454] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 1.
I20260811 22:20:53.152828 11088 leader_election.cc:302] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 3680fa72e4f3406cb973e80ef4fb8783, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:53.153005 11827 raft_consensus.cc:2781] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:53.153218 11827 raft_consensus.cc:686] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 LEADER]: Becoming Leader. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Running, Role: LEADER
I20260811 22:20:53.153496 11827 consensus_queue.cc:237] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.154634 11888 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: TxnStatusTablet state changed. Reason: New leader 359b52db26994a78994e5ae8f2cb26a8. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.154747 11888 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:53.155031 11893 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 22:20:53.155190 11893 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 22:20:53.155874 10946 catalog_manager.cc:5495] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:53.184870 11879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.181190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:42097 (local address 127.0.0.1:40178)
0811 22:20:53.181367 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.181383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:53.181491 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:53.181821 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:53.181823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:53.181834 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:53.181941 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.181946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.182697 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.182701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.183319 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.183328 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.183756 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.183760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.183849 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.183899 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.184572 (+ 673us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.184589 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 22:20:53.184654 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 22:20:53.184753 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260811 22:20:53.184872 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.181298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.29.3.65:42097)
0811 22:20:53.181457 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.181461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.181476 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:53.181646 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.181648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.181706 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.181792 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:53.182042 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.182583 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.183432 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.183646 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.183755 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.184037 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.184472 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.184576 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.184666 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.184723 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:20:53.188129 11441 raft_consensus.cc:1261] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Refusing update from remote peer 359b52db26994a78994e5ae8f2cb26a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.188238 11679 raft_consensus.cc:1261] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Refusing update from remote peer 359b52db26994a78994e5ae8f2cb26a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.188723 11888 consensus_queue.cc:1035] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.188922 11827 consensus_queue.cc:1035] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.190712 11845 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: TxnStatusTablet state changed. Reason: New leader 359b52db26994a78994e5ae8f2cb26a8. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.190836 11845 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:53.191402 11826 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: TxnStatusTablet state changed. Reason: New leader 359b52db26994a78994e5ae8f2cb26a8. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.191524 11826 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:53.192180 11827 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.192291 11827 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:53.194123 11845 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.194238 11845 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 3680fa72e4f3406cb973e80ef4fb8783: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:53.196475 11888 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.196592 11888 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 359b52db26994a78994e5ae8f2cb26a8: This TxnStatusTablet replica's current role is: LEADER
I20260811 22:20:53.197386 11826 tablet_replica.cc:439] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } } }
I20260811 22:20:53.197503 11826 tablet_replica.cc:442] T f3dd0722cce145b9b051f586d01fc61b P 50e7334048de452db11ecf7cd7610561: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 22:20:53.207378 11904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.204118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:42097 (local address 127.29.3.67:35383)
0811 22:20:53.204400 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.204416 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:53.204525 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:53.204750 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:53.204752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:53.204760 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:53.204863 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.204868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.205582 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.205585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.206207 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.206216 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.206679 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.206683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.206762 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.206807 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.207087 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.207093 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:20:53.207153 (+ 60us) client_negotiation.cc:240] Negotiation successful
0811 22:20:53.207243 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260811 22:20:53.207376 11815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.204244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:35383 (local address 127.29.3.65:42097)
0811 22:20:53.204409 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.204414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.204481 (+ 67us) server_negotiation.cc:407] Connection header received
0811 22:20:53.204615 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.204618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.204668 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.204727 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:53.204955 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.205474 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.206346 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.206567 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.206655 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.206932 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.207009 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.207066 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.207177 (+ 111us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.207231 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260811 22:20:53.214206 11209 raft_consensus.cc:1261] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Refusing update from remote peer 3680fa72e4f3406cb973e80ef4fb8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.214207 11441 raft_consensus.cc:1261] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Refusing update from remote peer 3680fa72e4f3406cb973e80ef4fb8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.214637 11845 consensus_queue.cc:1035] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.214814 11828 consensus_queue.cc:1035] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.237349 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.233729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:44573 (local address 127.29.3.67:42065)
0811 22:20:53.233842 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.233847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.234027 (+ 180us) server_negotiation.cc:407] Connection header received
0811 22:20:53.234140 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.234142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.234186 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.234253 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:53.234594 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.235261 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.236232 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.236575 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.236678 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.236922 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.237003 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.237067 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.237157 (+ 90us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.237218 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:20:53.237350 11912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.233631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:42065 (local address 127.29.3.65:44573)
0811 22:20:53.233946 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.233961 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:53.234066 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:53.234264 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:53.234267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:53.234274 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:53.234440 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.234447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.235382 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.235386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.236109 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.236120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.236699 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.236702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.236767 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.236811 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.237079 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.237084 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:20:53.237134 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 22:20:53.237202 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":97,"threads_started":1}
I20260811 22:20:53.261377 10946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:53.261534 10946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:53.263759 10946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Sending DeleteTablet for 3 replicas of tablet 43b0d878da874f8a9ec904b4f3137751
I20260811 22:20:53.264381 11168 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b0d878da874f8a9ec904b4f3137751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40176
I20260811 22:20:53.264415 11645 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b0d878da874f8a9ec904b4f3137751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38078
I20260811 22:20:53.264554 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b0d878da874f8a9ec904b4f3137751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39504
I20260811 22:20:53.264605 11855 tablet_replica.cc:330] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: stopping tablet replica
I20260811 22:20:53.264745 11855 raft_consensus.cc:2229] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:53.264765 11854 tablet_replica.cc:330] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: stopping tablet replica
I20260811 22:20:53.264770 11859 tablet_replica.cc:330] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: stopping tablet replica
I20260811 22:20:53.264871 11859 raft_consensus.cc:2229] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:53.264881 11854 raft_consensus.cc:2229] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:53.264967 11855 raft_consensus.cc:2258] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.264994 11854 raft_consensus.cc:2258] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.264981 11859 raft_consensus.cc:2258] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.265851 11854 ts_tablet_manager.cc:1905] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.265851 11855 ts_tablet_manager.cc:1905] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.265908 11859 ts_tablet_manager.cc:1905] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.268761 11854 ts_tablet_manager.cc:1918] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:53.268761 11855 ts_tablet_manager.cc:1918] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:53.268837 11854 log.cc:1196] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/43b0d878da874f8a9ec904b4f3137751
I20260811 22:20:53.268837 11855 log.cc:1196] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/43b0d878da874f8a9ec904b4f3137751
I20260811 22:20:53.268817 11859 ts_tablet_manager.cc:1918] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:53.268890 11859 log.cc:1196] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/43b0d878da874f8a9ec904b4f3137751
I20260811 22:20:53.269094 11854 ts_tablet_manager.cc:1939] T 43b0d878da874f8a9ec904b4f3137751 P 50e7334048de452db11ecf7cd7610561: Deleting consensus metadata
I20260811 22:20:53.269094 11855 ts_tablet_manager.cc:1939] T 43b0d878da874f8a9ec904b4f3137751 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting consensus metadata
I20260811 22:20:53.269137 11859 ts_tablet_manager.cc:1939] T 43b0d878da874f8a9ec904b4f3137751 P 359b52db26994a78994e5ae8f2cb26a8: Deleting consensus metadata
I20260811 22:20:53.269959 10922 catalog_manager.cc:4841] TS 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065): tablet 43b0d878da874f8a9ec904b4f3137751 (table test-table [id=806a3a08f55e4a60a1d68622cd1846b8]) successfully deleted
I20260811 22:20:53.272557 10918 catalog_manager.cc:4841] TS 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245): tablet 43b0d878da874f8a9ec904b4f3137751 (table test-table [id=806a3a08f55e4a60a1d68622cd1846b8]) successfully deleted
I20260811 22:20:53.274572 10922 catalog_manager.cc:4841] TS 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097): tablet 43b0d878da874f8a9ec904b4f3137751 (table test-table [id=806a3a08f55e4a60a1d68622cd1846b8]) successfully deleted
Aug 11 22:20:53 dist-test-slave-4tdj krb5kdc[10873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 22:20:53 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:53 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:20:53.299686 11027 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:53.293551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.29.3.126:42757)
0811 22:20:53.293676 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.293681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.293897 (+ 216us) server_negotiation.cc:407] Connection header received
0811 22:20:53.294048 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.294050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.294101 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.294176 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:53.294943 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.295469 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.296354 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.296572 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.296662 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.299552 (+ 2890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46864: BlockingRecv error: recv got EOF from 127.0.0.1:46864 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 22:20:53 dist-test-slave-4tdj krb5kdc[10873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM
I20260811 22:20:53.309340 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.301450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.29.3.126:42757)
0811 22:20:53.301560 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.301563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.301777 (+ 214us) server_negotiation.cc:407] Connection header received
0811 22:20:53.301879 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.301882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.301927 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.301994 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:53.302645 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.303151 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.303898 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.304123 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.304213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.306428 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:53.306431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:53.306434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:53.306463 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:53.307794 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:53.308181 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:53.308185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:53.308187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:53.308266 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:53.308513 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:53.308516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:53.308518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:53.308709 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:53.308829 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.309062 (+ 233us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.309180 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 22:20:53.312290 10946 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:53.312812 10946 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:53.317850 11168 tablet_service.cc:1467] Processing CreateTablet for tablet 183ab7dcac014c4cb3f11c051868385b (DEFAULT_TABLE table=test-table [id=2a1b22f0a86c4d4b99fb797e212cf3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:53.317898 11405 tablet_service.cc:1467] Processing CreateTablet for tablet 183ab7dcac014c4cb3f11c051868385b (DEFAULT_TABLE table=test-table [id=2a1b22f0a86c4d4b99fb797e212cf3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:53.317922 11645 tablet_service.cc:1467] Processing CreateTablet for tablet 183ab7dcac014c4cb3f11c051868385b (DEFAULT_TABLE table=test-table [id=2a1b22f0a86c4d4b99fb797e212cf3c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:53.318195 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183ab7dcac014c4cb3f11c051868385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:53.318195 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183ab7dcac014c4cb3f11c051868385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:53.318192 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 183ab7dcac014c4cb3f11c051868385b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:53.320761 11819 tablet_bootstrap.cc:492] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Bootstrap starting.
I20260811 22:20:53.321106 11820 tablet_bootstrap.cc:492] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Bootstrap starting.
I20260811 22:20:53.321601 11819 tablet_bootstrap.cc:654] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:53.321859 11820 tablet_bootstrap.cc:654] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:53.322136 11816 tablet_bootstrap.cc:492] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Bootstrap starting.
I20260811 22:20:53.322992 11820 tablet_bootstrap.cc:492] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: No bootstrap required, opened a new log
I20260811 22:20:53.323014 11816 tablet_bootstrap.cc:654] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:53.323099 11820 ts_tablet_manager.cc:1397] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:53.323514 11820 raft_consensus.cc:348] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.323639 11820 raft_consensus.cc:374] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:53.323679 11820 raft_consensus.cc:729] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Initialized, Role: FOLLOWER
I20260811 22:20:53.323801 11820 consensus_queue.cc:260] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.324054 11820 ts_tablet_manager.cc:1428] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:53.324944 11816 tablet_bootstrap.cc:492] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: No bootstrap required, opened a new log
I20260811 22:20:53.324944 11819 tablet_bootstrap.cc:492] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: No bootstrap required, opened a new log
I20260811 22:20:53.325037 11819 ts_tablet_manager.cc:1397] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260811 22:20:53.325037 11816 ts_tablet_manager.cc:1397] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 22:20:53.325464 11819 raft_consensus.cc:348] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.325464 11816 raft_consensus.cc:348] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.325582 11819 raft_consensus.cc:374] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:53.325582 11816 raft_consensus.cc:374] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:53.325623 11816 raft_consensus.cc:729] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7334048de452db11ecf7cd7610561, State: Initialized, Role: FOLLOWER
I20260811 22:20:53.325623 11819 raft_consensus.cc:729] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Initialized, Role: FOLLOWER
I20260811 22:20:53.325758 11819 consensus_queue.cc:260] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.325758 11816 consensus_queue.cc:260] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.326021 11816 ts_tablet_manager.cc:1428] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:53.326021 11819 ts_tablet_manager.cc:1428] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:53.639847 11831 raft_consensus.cc:482] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:53.640004 11831 raft_consensus.cc:504] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.640432 11831 leader_election.cc:288] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097), 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245)
I20260811 22:20:53.640805 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183ab7dcac014c4cb3f11c051868385b" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561" is_pre_election: true
I20260811 22:20:53.640836 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183ab7dcac014c4cb3f11c051868385b" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359b52db26994a78994e5ae8f2cb26a8" is_pre_election: true
I20260811 22:20:53.640944 11441 raft_consensus.cc:2454] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 0.
I20260811 22:20:53.640987 11209 raft_consensus.cc:2454] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 0.
I20260811 22:20:53.641278 11562 leader_election.cc:302] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 3680fa72e4f3406cb973e80ef4fb8783, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:53.641482 11831 raft_consensus.cc:2781] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:53.641546 11831 raft_consensus.cc:482] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:53.641571 11831 raft_consensus.cc:3037] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.642760 11831 raft_consensus.cc:504] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.643085 11831 leader_election.cc:288] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 election: Requested vote from peers 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097), 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245)
I20260811 22:20:53.643416 11209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183ab7dcac014c4cb3f11c051868385b" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359b52db26994a78994e5ae8f2cb26a8"
I20260811 22:20:53.643435 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "183ab7dcac014c4cb3f11c051868385b" candidate_uuid: "3680fa72e4f3406cb973e80ef4fb8783" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561"
I20260811 22:20:53.643536 11441 raft_consensus.cc:3037] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.643563 11209 raft_consensus.cc:3037] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.644413 11441 raft_consensus.cc:2454] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 1.
I20260811 22:20:53.644482 11209 raft_consensus.cc:2454] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3680fa72e4f3406cb973e80ef4fb8783 in term 1.
I20260811 22:20:53.644692 11559 leader_election.cc:302] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3680fa72e4f3406cb973e80ef4fb8783, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:53.644855 11831 raft_consensus.cc:2781] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:53.644944 11831 raft_consensus.cc:686] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 LEADER]: Becoming Leader. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Running, Role: LEADER
I20260811 22:20:53.645082 11831 consensus_queue.cc:237] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.646921 10947 catalog_manager.cc:5495] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "3680fa72e4f3406cb973e80ef4fb8783" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } health_report { overall_health: HEALTHY } } }
I20260811 22:20:53.695034 11810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.692013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.29.3.67:42065)
0811 22:20:53.692138 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.692142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.692157 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:20:53.692260 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.692263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.692319 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.692378 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:53.692629 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.693137 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.694003 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.694225 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.694340 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.694604 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.694678 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.694736 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.694842 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.694897 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 22:20:53.697042 11209 raft_consensus.cc:1261] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Refusing update from remote peer 3680fa72e4f3406cb973e80ef4fb8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.697436 11441 raft_consensus.cc:1261] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Refusing update from remote peer 3680fa72e4f3406cb973e80ef4fb8783: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.697451 11845 consensus_queue.cc:1035] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.697834 11845 consensus_queue.cc:1035] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.705029 10944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:53.705205 10944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:53.707311 10944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Sending DeleteTablet for 3 replicas of tablet 183ab7dcac014c4cb3f11c051868385b
I20260811 22:20:53.707860 11168 tablet_service.cc:1514] Processing DeleteTablet for tablet 183ab7dcac014c4cb3f11c051868385b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40176
I20260811 22:20:53.708050 11859 tablet_replica.cc:330] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: stopping tablet replica
I20260811 22:20:53.708191 11859 raft_consensus.cc:2229] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:53.708175 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet 183ab7dcac014c4cb3f11c051868385b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39504
I20260811 22:20:53.708329 11859 raft_consensus.cc:2258] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.708370 11854 tablet_replica.cc:330] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: stopping tablet replica
I20260811 22:20:53.708480 11854 raft_consensus.cc:2229] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:53.708484 11645 tablet_service.cc:1514] Processing DeleteTablet for tablet 183ab7dcac014c4cb3f11c051868385b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38078
I20260811 22:20:53.708616 11854 raft_consensus.cc:2258] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.708644 11855 tablet_replica.cc:330] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: stopping tablet replica
I20260811 22:20:53.708767 11855 raft_consensus.cc:2229] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:53.708950 11855 raft_consensus.cc:2258] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.709168 11859 ts_tablet_manager.cc:1905] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.709262 11854 ts_tablet_manager.cc:1905] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.709381 10944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:20:53.709769 10944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:20:53.711380 11859 ts_tablet_manager.cc:1918] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:53.711439 11859 log.cc:1196] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/183ab7dcac014c4cb3f11c051868385b
I20260811 22:20:53.711651 11859 ts_tablet_manager.cc:1939] T 183ab7dcac014c4cb3f11c051868385b P 359b52db26994a78994e5ae8f2cb26a8: Deleting consensus metadata
I20260811 22:20:53.711685 11854 ts_tablet_manager.cc:1918] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:53.711763 11854 log.cc:1196] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/183ab7dcac014c4cb3f11c051868385b
I20260811 22:20:53.712522 10922 catalog_manager.cc:4841] TS 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097): tablet 183ab7dcac014c4cb3f11c051868385b (table test-table [id=2a1b22f0a86c4d4b99fb797e212cf3c8]) successfully deleted
I20260811 22:20:53.712966 11854 ts_tablet_manager.cc:1939] T 183ab7dcac014c4cb3f11c051868385b P 50e7334048de452db11ecf7cd7610561: Deleting consensus metadata
I20260811 22:20:53.713641 11855 ts_tablet_manager.cc:1905] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.714533 11405 tablet_service.cc:1467] Processing CreateTablet for tablet 8432d5fea47840d1b14d32ee605b5a37 (DEFAULT_TABLE table=test-table [id=d5014047c70044189414bcb83a44e0cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:53.714538 11168 tablet_service.cc:1467] Processing CreateTablet for tablet 8432d5fea47840d1b14d32ee605b5a37 (DEFAULT_TABLE table=test-table [id=d5014047c70044189414bcb83a44e0cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:53.714795 11168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8432d5fea47840d1b14d32ee605b5a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:53.715812 11855 ts_tablet_manager.cc:1918] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:20:53.715876 11855 log.cc:1196] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/183ab7dcac014c4cb3f11c051868385b
I20260811 22:20:53.716040 11855 ts_tablet_manager.cc:1939] T 183ab7dcac014c4cb3f11c051868385b P 3680fa72e4f3406cb973e80ef4fb8783: Deleting consensus metadata
I20260811 22:20:53.716356 11405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8432d5fea47840d1b14d32ee605b5a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:53.716725 10918 catalog_manager.cc:4841] TS 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245): tablet 183ab7dcac014c4cb3f11c051868385b (table test-table [id=2a1b22f0a86c4d4b99fb797e212cf3c8]) successfully deleted
I20260811 22:20:53.716735 10922 catalog_manager.cc:4841] TS 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065): tablet 183ab7dcac014c4cb3f11c051868385b (table test-table [id=2a1b22f0a86c4d4b99fb797e212cf3c8]) successfully deleted
I20260811 22:20:53.717000 11645 tablet_service.cc:1467] Processing CreateTablet for tablet 8432d5fea47840d1b14d32ee605b5a37 (DEFAULT_TABLE table=test-table [id=d5014047c70044189414bcb83a44e0cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:20:53.717242 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8432d5fea47840d1b14d32ee605b5a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:53.718569 11819 tablet_bootstrap.cc:492] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8: Bootstrap starting.
I20260811 22:20:53.718616 11816 tablet_bootstrap.cc:492] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Bootstrap starting.
I20260811 22:20:53.719302 11819 tablet_bootstrap.cc:654] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:53.719409 11820 tablet_bootstrap.cc:492] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Bootstrap starting.
I20260811 22:20:53.719460 11816 tablet_bootstrap.cc:654] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:53.720222 11820 tablet_bootstrap.cc:654] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:53.720631 11816 tablet_bootstrap.cc:492] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: No bootstrap required, opened a new log
I20260811 22:20:53.720736 11816 ts_tablet_manager.cc:1397] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:53.721195 11816 raft_consensus.cc:348] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.721324 11816 raft_consensus.cc:374] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:53.721350 11820 tablet_bootstrap.cc:492] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: No bootstrap required, opened a new log
I20260811 22:20:53.721369 11816 raft_consensus.cc:729] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50e7334048de452db11ecf7cd7610561, State: Initialized, Role: FOLLOWER
I20260811 22:20:53.721431 11820 ts_tablet_manager.cc:1397] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 22:20:53.721506 11816 consensus_queue.cc:260] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.721789 11816 ts_tablet_manager.cc:1428] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:20:53.721812 11820 raft_consensus.cc:348] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.721915 11820 raft_consensus.cc:374] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:53.721946 11820 raft_consensus.cc:729] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3680fa72e4f3406cb973e80ef4fb8783, State: Initialized, Role: FOLLOWER
I20260811 22:20:53.722060 11820 consensus_queue.cc:260] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.722378 11820 ts_tablet_manager.cc:1428] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 22:20:53.722455 11819 tablet_bootstrap.cc:492] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8: No bootstrap required, opened a new log
I20260811 22:20:53.722564 11819 ts_tablet_manager.cc:1397] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260811 22:20:53.722980 11819 raft_consensus.cc:348] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.723078 11819 raft_consensus.cc:374] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:53.723099 11819 raft_consensus.cc:729] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Initialized, Role: FOLLOWER
I20260811 22:20:53.723222 11819 consensus_queue.cc:260] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.723469 11819 ts_tablet_manager.cc:1428] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 22:20:53.832738 11827 raft_consensus.cc:482] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:20:53.832890 11827 raft_consensus.cc:504] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.833331 11827 leader_election.cc:288] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245), 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065)
I20260811 22:20:53.833830 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8432d5fea47840d1b14d32ee605b5a37" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561" is_pre_election: true
I20260811 22:20:53.833830 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8432d5fea47840d1b14d32ee605b5a37" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3680fa72e4f3406cb973e80ef4fb8783" is_pre_election: true
I20260811 22:20:53.833992 11679 raft_consensus.cc:2454] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 0.
I20260811 22:20:53.833992 11441 raft_consensus.cc:2454] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 0.
I20260811 22:20:53.834332 11091 leader_election.cc:302] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 3680fa72e4f3406cb973e80ef4fb8783; no voters:
I20260811 22:20:53.834518 11827 raft_consensus.cc:2781] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:20:53.834590 11827 raft_consensus.cc:482] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:20:53.834617 11827 raft_consensus.cc:3037] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.835629 11827 raft_consensus.cc:504] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.835995 11827 leader_election.cc:288] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 election: Requested vote from peers 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245), 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065)
I20260811 22:20:53.836376 11679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8432d5fea47840d1b14d32ee605b5a37" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3680fa72e4f3406cb973e80ef4fb8783"
I20260811 22:20:53.836493 11679 raft_consensus.cc:3037] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.836437 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8432d5fea47840d1b14d32ee605b5a37" candidate_uuid: "359b52db26994a78994e5ae8f2cb26a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50e7334048de452db11ecf7cd7610561"
I20260811 22:20:53.836606 11441 raft_consensus.cc:3037] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:53.837554 11679 raft_consensus.cc:2454] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 1.
I20260811 22:20:53.837564 11441 raft_consensus.cc:2454] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359b52db26994a78994e5ae8f2cb26a8 in term 1.
I20260811 22:20:53.837844 11091 leader_election.cc:302] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 359b52db26994a78994e5ae8f2cb26a8, 3680fa72e4f3406cb973e80ef4fb8783, 50e7334048de452db11ecf7cd7610561; no voters:
I20260811 22:20:53.838008 11827 raft_consensus.cc:2781] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:53.838207 11827 raft_consensus.cc:686] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 LEADER]: Becoming Leader. State: Replica: 359b52db26994a78994e5ae8f2cb26a8, State: Running, Role: LEADER
I20260811 22:20:53.838416 11827 consensus_queue.cc:237] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } }
I20260811 22:20:53.840066 10944 catalog_manager.cc:5495] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "359b52db26994a78994e5ae8f2cb26a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "359b52db26994a78994e5ae8f2cb26a8" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 42097 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 } health_report { overall_health: UNKNOWN } } }
I20260811 22:20:53.884505 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.881620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.29.3.65:42097)
0811 22:20:53.881836 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.881839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.881851 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:53.881891 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.881893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.881934 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.882006 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:53.882182 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.882765 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.883595 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.883863 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.883951 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.884138 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.884214 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.884266 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.884329 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.884377 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260811 22:20:53.893199 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.890502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:51527 (local address 127.29.3.65:42097)
0811 22:20:53.890633 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:53.890638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:53.890751 (+ 113us) server_negotiation.cc:407] Connection header received
0811 22:20:53.890886 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:53.890889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:53.890935 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:53.891000 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:20:53.891187 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.891674 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.892419 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.892620 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.892695 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.892856 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.892917 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.892964 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:53.893029 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 22:20:53.893073 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:53.893227 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.890439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:42097 (local address 127.29.3.65:51527)
0811 22:20:53.890677 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.890689 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:53.890780 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:53.891011 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:53.891013 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:53.891018 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 22:20:53.891111 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.891115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.891748 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.891750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.892339 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:53.892346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:53.892702 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:53.892705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:53.892760 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:53.892784 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 22:20:53.892973 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 22:20:53.892977 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:20:53.893013 (+ 36us) client_negotiation.cc:240] Negotiation successful
0811 22:20:53.893075 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260811 22:20:53.898749 11679 raft_consensus.cc:1261] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Refusing update from remote peer 359b52db26994a78994e5ae8f2cb26a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.898746 11441 raft_consensus.cc:1261] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Refusing update from remote peer 359b52db26994a78994e5ae8f2cb26a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:20:53.899134 11950 consensus_queue.cc:1035] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50e7334048de452db11ecf7cd7610561" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.899283 11953 consensus_queue.cc:1035] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3680fa72e4f3406cb973e80ef4fb8783" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:20:53.931519 10946 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:53.931704 10946 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:20:53.933674 10946 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 008e365a4be04ae0b0839ea402e04d70: Sending DeleteTablet for 3 replicas of tablet 8432d5fea47840d1b14d32ee605b5a37
I20260811 22:20:53.934265 11645 tablet_service.cc:1514] Processing DeleteTablet for tablet 8432d5fea47840d1b14d32ee605b5a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38078
I20260811 22:20:53.934265 11405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8432d5fea47840d1b14d32ee605b5a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39504
I20260811 22:20:53.934465 11855 tablet_replica.cc:330] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: stopping tablet replica
I20260811 22:20:53.934465 11854 tablet_replica.cc:330] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: stopping tablet replica
I20260811 22:20:53.934566 11854 raft_consensus.cc:2229] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:53.934566 11855 raft_consensus.cc:2229] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:20:53.934667 11855 raft_consensus.cc:2258] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.934681 11854 raft_consensus.cc:2258] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.935307 11168 tablet_service.cc:1514] Processing DeleteTablet for tablet 8432d5fea47840d1b14d32ee605b5a37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:20:53 UTC) from {username='oryx', principal='oryx/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40176
I20260811 22:20:53.935468 11859 tablet_replica.cc:330] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8: stopping tablet replica
I20260811 22:20:53.935557 11855 ts_tablet_manager.cc:1905] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.935570 11859 raft_consensus.cc:2229] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:20:53.935853 11859 raft_consensus.cc:2258] T 8432d5fea47840d1b14d32ee605b5a37 P 359b52db26994a78994e5ae8f2cb26a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:20:53.936158 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 11063
I20260811 22:20:53.937930 11855 ts_tablet_manager.cc:1918] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:53.938004 11855 log.cc:1196] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/8432d5fea47840d1b14d32ee605b5a37
I20260811 22:20:53.938226 11855 ts_tablet_manager.cc:1939] T 8432d5fea47840d1b14d32ee605b5a37 P 3680fa72e4f3406cb973e80ef4fb8783: Deleting consensus metadata
I20260811 22:20:53.939869 10922 catalog_manager.cc:4841] TS 3680fa72e4f3406cb973e80ef4fb8783 (127.29.3.67:42065): tablet 8432d5fea47840d1b14d32ee605b5a37 (table test-table [id=d5014047c70044189414bcb83a44e0cc]) successfully deleted
I20260811 22:20:53.946851 11854 ts_tablet_manager.cc:1905] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:20:53.948743 11854 ts_tablet_manager.cc:1918] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 22:20:53.948812 11854 log.cc:1196] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/8432d5fea47840d1b14d32ee605b5a37
I20260811 22:20:53.949023 11854 ts_tablet_manager.cc:1939] T 8432d5fea47840d1b14d32ee605b5a37 P 50e7334048de452db11ecf7cd7610561: Deleting consensus metadata
W20260811 22:20:53.949422 10922 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:42097 (error 108)
I20260811 22:20:53.949705 10918 catalog_manager.cc:4841] TS 50e7334048de452db11ecf7cd7610561 (127.29.3.66:46245): tablet 8432d5fea47840d1b14d32ee605b5a37 (table test-table [id=d5014047c70044189414bcb83a44e0cc]) successfully deleted
I20260811 22:20:53.950419 11968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:53.950062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:42097 (local address 127.0.0.1:40192)
0811 22:20:53.950248 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:53.950338 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:42097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":53,"threads_started":1}
I20260811 22:20:53.950448 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 11293
W20260811 22:20:53.950554 10922 catalog_manager.cc:4584] TS 359b52db26994a78994e5ae8f2cb26a8 (127.29.3.65:42097): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8432d5fea47840d1b14d32ee605b5a37: Network error: Client connection negotiation failed: client connection to 127.29.3.65:42097: connect: Connection refused (error 111)
I20260811 22:20:53.964517 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 11534
I20260811 22:20:53.974150 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 10891
2026-08-11T22:20:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3776 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:54 dist-test-slave-4tdj krb5kdc[11972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:54 dist-test-slave-4tdj krb5kdc[11972](info): set up 2 sockets
Aug 11 22:20:54 dist-test-slave-4tdj krb5kdc[11972](info): commencing operation
krb5kdc: starting...
W20260811 22:20:55.162688 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.156s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:55 dist-test-slave-4tdj krb5kdc[11972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:55.289170 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35873
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:36933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:55.402697 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:55.402990 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:55.403031 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:55.407117 11986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:55.407182 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:55.407202 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:55.407223 11986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:55.407238 11986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:55.412638 11986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:36933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:55.413769 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:55.415125 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:55.423713 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:55.423709 12004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:55.423781 11986 server_base.cc:1034] running on GCE node
W20260811 22:20:55.423765 12002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:55.424474 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:55.425596 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:55.426746 11986 hybrid_clock.cc:648] HybridClock initialized: now 1786486855426723 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T22:20:55Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1453 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:20:55 dist-test-slave-4tdj krb5kdc[12012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:20:55 dist-test-slave-4tdj krb5kdc[12012](info): set up 2 sockets
Aug 11 22:20:55 dist-test-slave-4tdj krb5kdc[12012](info): commencing operation
krb5kdc: starting...
W20260811 22:20:56.623497 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:20:56 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:20:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:20:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:20:56.748664 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40143
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:56.861037 12034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:56.861310 12034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:56.861353 12034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:56.865646 12034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:56.865718 12034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:56.865737 12034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:56.865753 12034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:56.865767 12034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:56.871236 12034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:56.872284 12034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:56.873253 12034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:56.882019 12047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:56.882019 12050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:56.882123 12034 server_base.cc:1034] running on GCE node
W20260811 22:20:56.882093 12053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:56.882961 12034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:56.884083 12034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:56.885252 12034 hybrid_clock.cc:648] HybridClock initialized: now 1786486856885239 us; error 29 us; skew 500 ppm
Aug 11 22:20:56 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486856, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:56.887965 12034 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:56.888854 12034 webserver.cc:460] Webserver started at http://127.29.3.126:33265/ using document root <none> and password file <none>
I20260811 22:20:56.889381 12034 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:56.889439 12034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:56.889612 12034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:56.891667 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "492f808e1ace408393607287837bccdd"
format_stamp: "Formatted at 2026-08-11 22:20:56 on dist-test-slave-4tdj"
server_key: "3d6b482526b75d816598666ab8b05e52"
server_key_iv: "b0656b28624380aaad338961e8fc62a7"
server_key_version: "encryptionkey@0"
I20260811 22:20:56.892175 12034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "492f808e1ace408393607287837bccdd"
format_stamp: "Formatted at 2026-08-11 22:20:56 on dist-test-slave-4tdj"
server_key: "3d6b482526b75d816598666ab8b05e52"
server_key_iv: "b0656b28624380aaad338961e8fc62a7"
server_key_version: "encryptionkey@0"
I20260811 22:20:56.896111 12034 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:56.898583 12065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:56.899760 12034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:20:56.899870 12034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "492f808e1ace408393607287837bccdd"
format_stamp: "Formatted at 2026-08-11 22:20:56 on dist-test-slave-4tdj"
server_key: "3d6b482526b75d816598666ab8b05e52"
server_key_iv: "b0656b28624380aaad338961e8fc62a7"
server_key_version: "encryptionkey@0"
I20260811 22:20:56.899976 12034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:56.917158 12034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:56.920396 12034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:56.920565 12034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:56.928078 12034 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:40143
I20260811 22:20:56.928100 12150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:40143 every 8 connection(s)
I20260811 22:20:56.929081 12034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:56.932072 12151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:20:56.935647 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 12034
I20260811 22:20:56.935789 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:20:56.936087 29709 external_mini_cluster.cc:1428] Setting key 1741620f0c9d77ab4fb24c40929a7478
I20260811 22:20:56.937948 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Bootstrap starting.
I20260811 22:20:56.940678 12151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Neither blocks nor log segments found. Creating new log.
I20260811 22:20:56.941558 12151 log.cc:824] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:56.943776 12151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: No bootstrap required, opened a new log
Aug 11 22:20:56 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:56.947703 12151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:56.947968 12151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:20:56.948035 12151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492f808e1ace408393607287837bccdd, State: Initialized, Role: FOLLOWER
I20260811 22:20:56.948526 12151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:56.948730 12151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:56.948800 12151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:56.948902 12151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:20:56.949734 12151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:56.950071 12151 leader_election.cc:302] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 492f808e1ace408393607287837bccdd; no voters:
I20260811 22:20:56.950379 12151 leader_election.cc:288] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:20:56.950449 12156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:20:56.950644 12156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 LEADER]: Becoming Leader. State: Replica: 492f808e1ace408393607287837bccdd, State: Running, Role: LEADER
I20260811 22:20:56.950891 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:56.937518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.29.3.126:40143)
0811 22:20:56.937931 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:56.937951 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:56.937996 (+ 45us) server_negotiation.cc:407] Connection header received
0811 22:20:56.938827 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:56.938851 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:56.939288 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:56.939652 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:56.940277 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:56.941747 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:56.942770 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:56.943432 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:56.943599 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:56.945847 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:56.945853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:56.945870 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:56.945902 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:56.947778 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:56.948243 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:56.948249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:56.948255 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:56.948389 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:56.948682 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:56.948685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:56.948687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:56.949168 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:56.949376 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:56.949652 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 22:20:56.949958 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":106,"threads_started":1}
I20260811 22:20:56.950989 12156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:56.951658 12151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:56.953471 12158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "492f808e1ace408393607287837bccdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } } }
I20260811 22:20:56.953601 12158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:56.953922 12160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492f808e1ace408393607287837bccdd. Latest consensus state: current_term: 1 leader_uuid: "492f808e1ace408393607287837bccdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } } }
I20260811 22:20:56.954016 12160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: This master's current role is: LEADER
W20260811 22:20:56.956176 12185 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:20:56.956239 12185 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:20:56.956310 12188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:56.957055 12188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:56.962751 12188 catalog_manager.cc:1347] Generated new cluster ID: 764debb66b934a079d4bdf4c82d84f76
I20260811 22:20:56.962826 12188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:56.991593 12188 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:20:56.992921 12188 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:57.002226 12188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Generated new TSK 0
I20260811 22:20:57.003067 12188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:20:57.041793 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40143
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:57.165829 12198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:57.166090 12198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:57.166129 12198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:57.170066 12198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:57.170127 12198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:57.170184 12198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:57.175369 12198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:57.176385 12198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:57.177372 12198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:57.187021 12212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.187131 12198 server_base.cc:1034] running on GCE node
W20260811 22:20:57.187100 12210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:57.187309 12217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.187726 12198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:57.188364 12198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:57.189515 12198 hybrid_clock.cc:648] HybridClock initialized: now 1786486857189491 us; error 37 us; skew 500 ppm
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:57.192286 12198 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:20:57.193178 12198 webserver.cc:460] Webserver started at http://127.29.3.65:36827/ using document root <none> and password file <none>
I20260811 22:20:57.193701 12198 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:57.193758 12198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:57.193928 12198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:57.196072 12198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "9a221ff2ff0241948219baecfe469507"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "561c92605e154a8db1ca0ab98e406119"
server_key_iv: "374dfc7b8c5d91555742429f4184da48"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.196604 12198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "9a221ff2ff0241948219baecfe469507"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "561c92605e154a8db1ca0ab98e406119"
server_key_iv: "374dfc7b8c5d91555742429f4184da48"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.200435 12198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 22:20:57.202961 12225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.204324 12198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:20:57.204452 12198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "9a221ff2ff0241948219baecfe469507"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "561c92605e154a8db1ca0ab98e406119"
server_key_iv: "374dfc7b8c5d91555742429f4184da48"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.204586 12198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:57.239802 12198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:57.243209 12198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:57.243397 12198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:57.244032 12198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:57.244927 12198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:57.244978 12198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.245024 12198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:57.245054 12198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.254006 12198 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:44853
I20260811 22:20:57.254030 12418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:44853 every 8 connection(s)
I20260811 22:20:57.254963 12198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:57.260555 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 12198
I20260811 22:20:57.260677 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:20:57.260962 29709 external_mini_cluster.cc:1428] Setting key 7c36b84a743f60a79be02093a46a4b33
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:57.269222 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:57.256771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:33941 (local address 127.29.3.126:40143)
0811 22:20:57.256908 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:57.256911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:57.257760 (+ 849us) server_negotiation.cc:407] Connection header received
0811 22:20:57.259092 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:57.259095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:57.259146 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:57.259225 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:57.260417 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.261025 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.262228 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.262580 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.262693 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:57.265556 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:57.265560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:57.265562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:57.265588 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:57.266961 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:57.267616 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:57.267619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:57.267621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:57.267704 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:57.268081 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:57.268083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:57.268084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:57.268257 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:57.268328 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:57.268985 (+ 657us) server_negotiation.cc:299] Negotiation successful
0811 22:20:57.269095 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:20:57.270560 12424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:57.257143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:33941)
0811 22:20:57.257606 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:57.257649 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:57.258878 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:57.259387 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:57.259394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:57.259749 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:57.260241 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:57.260256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.261143 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.261147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:57.262115 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:57.262126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.262728 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.262731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:57.262883 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:57.263534 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:57.263559 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:57.265371 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:57.267092 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:57.267100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:57.267120 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:57.267497 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:57.267792 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:57.267796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:57.267798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:57.267994 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:57.268356 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:57.268363 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:57.268724 (+ 361us) client_negotiation.cc:769] Sending connection context
0811 22:20:57.268974 (+ 250us) client_negotiation.cc:240] Negotiation successful
0811 22:20:57.269375 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:20:57.271878 12422 heartbeater.cc:347] Connected to a master server at 127.29.3.126:40143
I20260811 22:20:57.272114 12422 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:57.272629 12422 heartbeater.cc:511] Master 127.29.3.126:40143 requested a full tablet report, sending...
I20260811 22:20:57.274188 12095 ts_manager.cc:194] Registered new tserver with Master: 9a221ff2ff0241948219baecfe469507 (127.29.3.65:44853)
I20260811 22:20:57.275421 12095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:33941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:20:57.292786 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40143
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:57.405428 12429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:57.405702 12429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:57.405761 12429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:57.409798 12429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:57.409863 12429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:57.409942 12429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:57.415244 12429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:57.416318 12429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:57.417335 12429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:57.426710 12435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:57.426730 12436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.426812 12429 server_base.cc:1034] running on GCE node
W20260811 22:20:57.426712 12438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.427330 12429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:57.427927 12429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:57.429071 12429 hybrid_clock.cc:648] HybridClock initialized: now 1786486857429045 us; error 38 us; skew 500 ppm
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:57.431753 12429 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:20:57.432658 12429 webserver.cc:460] Webserver started at http://127.29.3.66:36433/ using document root <none> and password file <none>
I20260811 22:20:57.433204 12429 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:57.433284 12429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:57.433494 12429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:57.435575 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "9d393d278e984b93b219e3eb63cd2528"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "730f7a06089de34a5d09702987748d7d"
server_key_iv: "99f918b0769aeaa591e2708ed5f41215"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.436120 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "9d393d278e984b93b219e3eb63cd2528"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "730f7a06089de34a5d09702987748d7d"
server_key_iv: "99f918b0769aeaa591e2708ed5f41215"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.439921 12429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:20:57.442451 12448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.443672 12429 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:57.443783 12429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "9d393d278e984b93b219e3eb63cd2528"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "730f7a06089de34a5d09702987748d7d"
server_key_iv: "99f918b0769aeaa591e2708ed5f41215"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.443872 12429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:57.457948 12429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:57.460889 12429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:57.461050 12429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:57.461617 12429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:57.462467 12429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:57.462518 12429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.462567 12429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:57.462586 12429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.471108 12429 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:44679
I20260811 22:20:57.471134 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:44679 every 8 connection(s)
I20260811 22:20:57.472029 12429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:57.479645 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 12429
I20260811 22:20:57.479786 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:20:57.480042 29709 external_mini_cluster.cc:1428] Setting key 5925502c22b7c96077235a03ad5ea757
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:57.485472 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:57.473892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:43903 (local address 127.29.3.126:40143)
0811 22:20:57.474033 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:57.474037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:57.474954 (+ 917us) server_negotiation.cc:407] Connection header received
0811 22:20:57.476283 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:57.476286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:57.476347 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:57.476418 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:57.477618 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.478140 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.479316 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.479543 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.479634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:57.482155 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:57.482165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:57.482167 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:57.482192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:57.483494 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:57.483976 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:57.483979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:57.483981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:57.484061 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:57.484379 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:57.484382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:57.484384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:57.484569 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:57.484656 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:57.485152 (+ 496us) server_negotiation.cc:299] Negotiation successful
0811 22:20:57.485306 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:20:57.486372 12653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:57.474324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:43903)
0811 22:20:57.474799 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:57.474844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:57.476059 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:57.476602 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:57.476610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:57.476976 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:57.477464 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:57.477478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.478273 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.478276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:57.479183 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:57.479193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.479696 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.479700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:57.479891 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:57.480469 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:57.480486 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:57.481993 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:57.483603 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:57.483609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:57.483625 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:57.483881 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:57.484145 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:57.484148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:57.484150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:57.484289 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:57.484679 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:57.484685 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:57.484923 (+ 238us) client_negotiation.cc:769] Sending connection context
0811 22:20:57.485092 (+ 169us) client_negotiation.cc:240] Negotiation successful
0811 22:20:57.485432 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":125,"threads_started":1}
I20260811 22:20:57.487594 12647 heartbeater.cc:347] Connected to a master server at 127.29.3.126:40143
I20260811 22:20:57.487854 12647 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:57.488334 12647 heartbeater.cc:511] Master 127.29.3.126:40143 requested a full tablet report, sending...
I20260811 22:20:57.489387 12095 ts_manager.cc:194] Registered new tserver with Master: 9d393d278e984b93b219e3eb63cd2528 (127.29.3.66:44679)
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:20:57.489991 12095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:43903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:20:57.506268 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40143
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:57.619148 12657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:57.619534 12657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:57.619614 12657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:57.625077 12657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:57.625149 12657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:57.625249 12657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:57.631199 12657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:57.632298 12657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:57.633333 12657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:57.643167 12672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.643340 12657 server_base.cc:1034] running on GCE node
W20260811 22:20:57.643198 12669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:57.643179 12668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.643896 12657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:57.644609 12657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:57.645781 12657 hybrid_clock.cc:648] HybridClock initialized: now 1786486857645761 us; error 40 us; skew 500 ppm
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:57.648635 12657 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:20:57.649487 12657 webserver.cc:460] Webserver started at http://127.29.3.67:42333/ using document root <none> and password file <none>
I20260811 22:20:57.650115 12657 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:57.650182 12657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:57.650411 12657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:20:57.652554 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "71747473336b4def8d61949214ef5480"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "7a03ac3b1a7a16ee19835f38b8b8322f"
server_key_iv: "7c9e378c18fe738939eccc3e74903344"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.653168 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "71747473336b4def8d61949214ef5480"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "7a03ac3b1a7a16ee19835f38b8b8322f"
server_key_iv: "7c9e378c18fe738939eccc3e74903344"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.656915 12657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:20:57.659410 12684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.660629 12657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:20:57.660740 12657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "71747473336b4def8d61949214ef5480"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "7a03ac3b1a7a16ee19835f38b8b8322f"
server_key_iv: "7c9e378c18fe738939eccc3e74903344"
server_key_version: "encryptionkey@0"
I20260811 22:20:57.660836 12657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:57.678797 12657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:57.682067 12657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:57.682262 12657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:57.682926 12657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:57.683797 12657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:57.683847 12657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.683900 12657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:57.683928 12657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:57.692603 12657 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:40035
I20260811 22:20:57.692629 12896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:40035 every 8 connection(s)
I20260811 22:20:57.693548 12657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:20:57.696070 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 12657
I20260811 22:20:57.696219 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:20:57.696563 29709 external_mini_cluster.cc:1428] Setting key 5029861130503cc433a9751292921805
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:20:57.708487 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:57.695463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:59745 (local address 127.29.3.126:40143)
0811 22:20:57.695638 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:57.695642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:57.696572 (+ 930us) server_negotiation.cc:407] Connection header received
0811 22:20:57.698184 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:57.698188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:57.698253 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:57.698380 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:57.699622 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.700217 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.701902 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.702250 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.702401 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:57.705261 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:57.705264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:57.705266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:57.705294 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:57.706486 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:57.706965 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:57.706968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:57.706969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:57.707044 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:57.707366 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:57.707369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:57.707371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:57.707576 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:57.707666 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:57.708214 (+ 548us) server_negotiation.cc:299] Negotiation successful
0811 22:20:57.708331 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 22:20:57.709515 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:57.695831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:59745)
0811 22:20:57.696388 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:57.696450 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:57.697961 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:57.698531 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:57.698541 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:57.698981 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:57.699497 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:57.699512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.700371 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.700374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:57.701768 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:57.701781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:57.702598 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:57.702602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:57.702811 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:57.703474 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:57.703499 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:57.705072 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:20:57.706597 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:57.706603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:57.706616 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:57.706880 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:57.707116 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:20:57.707118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:20:57.707120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:20:57.707261 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:20:57.707723 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:20:57.707729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:20:57.708003 (+ 274us) client_negotiation.cc:769] Sending connection context
0811 22:20:57.708177 (+ 174us) client_negotiation.cc:240] Negotiation successful
0811 22:20:57.708518 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":127,"threads_started":1}
I20260811 22:20:57.710723 12898 heartbeater.cc:347] Connected to a master server at 127.29.3.126:40143
I20260811 22:20:57.710973 12898 heartbeater.cc:464] Registering TS with master...
I20260811 22:20:57.711485 12898 heartbeater.cc:511] Master 127.29.3.126:40143 requested a full tablet report, sending...
I20260811 22:20:57.712589 12094 ts_manager.cc:194] Registered new tserver with Master: 71747473336b4def8d61949214ef5480 (127.29.3.67:40035)
I20260811 22:20:57.713160 12094 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:59745
I20260811 22:20:57.722525 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab.
Entry for principal oryx/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab.
WARNING: no policy specified for oryx/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab.
Entry for principal oryx/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab.
WARNING: no policy specified for oryx/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab.
Entry for principal oryx/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab.
I20260811 22:20:57.794996 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 12198
I20260811 22:20:57.809782 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 12429
I20260811 22:20:57.830684 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 12657
I20260811 22:20:57.845299 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 12034
I20260811 22:20:57.861181 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40143
--webserver_interface=127.29.3.126
--webserver_port=33265
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:40143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:20:57.976013 12910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:57.976297 12910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:57.976341 12910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:57.980377 12910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:20:57.980446 12910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:57.980465 12910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:57.980480 12910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:20:57.980499 12910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:20:57.985819 12910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:40143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=33265
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:57.986909 12910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:57.987926 12910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:57.996417 12926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:57.996429 12919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.996469 12910 server_base.cc:1034] running on GCE node
W20260811 22:20:57.996411 12923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:57.997272 12910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:57.998376 12910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:57.999543 12910 hybrid_clock.cc:648] HybridClock initialized: now 1786486857999525 us; error 35 us; skew 500 ppm
Aug 11 22:20:57 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:58.002199 12910 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:20:58.003106 12910 webserver.cc:460] Webserver started at http://127.29.3.126:33265/ using document root <none> and password file <none>
I20260811 22:20:58.003665 12910 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:58.003751 12910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:58.007622 12910 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:58.009722 12939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.010962 12910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:58.011093 12910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "492f808e1ace408393607287837bccdd"
format_stamp: "Formatted at 2026-08-11 22:20:56 on dist-test-slave-4tdj"
server_key: "3d6b482526b75d816598666ab8b05e52"
server_key_iv: "b0656b28624380aaad338961e8fc62a7"
server_key_version: "encryptionkey@0"
I20260811 22:20:58.011636 12910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:58.043207 12910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:58.046164 12910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:58.046386 12910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:58.053604 12910 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:40143
I20260811 22:20:58.053661 13025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:40143 every 8 connection(s)
I20260811 22:20:58.054761 12910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:20:58.058002 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 12910
I20260811 22:20:58.058465 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:44853
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:40143
--webserver_port=36827
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:20:58.059293 13026 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:20:58.061092 13026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Bootstrap starting.
I20260811 22:20:58.084232 13026 log.cc:824] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Log is configured to *not* fsync() on all Append() calls
I20260811 22:20:58.093041 13026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:20:58.093518 13026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Bootstrap complete.
I20260811 22:20:58.099254 13026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:58.099789 13026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492f808e1ace408393607287837bccdd, State: Initialized, Role: FOLLOWER
I20260811 22:20:58.100374 13026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:58.100541 13026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:20:58.100615 13026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:20:58.100713 13026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:20:58.103480 13026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:58.103907 13026 leader_election.cc:302] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 492f808e1ace408393607287837bccdd; no voters:
I20260811 22:20:58.104291 13026 leader_election.cc:288] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:20:58.104415 13031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:20:58.104686 13031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [term 2 LEADER]: Becoming Leader. State: Replica: 492f808e1ace408393607287837bccdd, State: Running, Role: LEADER
I20260811 22:20:58.105042 13031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } }
I20260811 22:20:58.105676 13026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:20:58.106526 13036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 492f808e1ace408393607287837bccdd. Latest consensus state: current_term: 2 leader_uuid: "492f808e1ace408393607287837bccdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } } }
I20260811 22:20:58.106526 13034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "492f808e1ace408393607287837bccdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "492f808e1ace408393607287837bccdd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40143 } } }
I20260811 22:20:58.106686 13036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:58.106686 13034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd [sys.catalog]: This master's current role is: LEADER
I20260811 22:20:58.107223 13048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:20:58.111063 13048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:20:58.111912 13048 catalog_manager.cc:1259] Loaded cluster ID: 764debb66b934a079d4bdf4c82d84f76
I20260811 22:20:58.111970 13048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:20:58.113699 13048 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:20:58.114351 13048 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 492f808e1ace408393607287837bccdd: Loaded TSK: 0
I20260811 22:20:58.115350 13048 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:20:58.201941 13028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:58.202224 13028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:58.202311 13028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:58.206547 13028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:58.206617 13028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:58.206745 13028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:20:58.212285 13028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.65.keytab
--principal=oryx/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:44853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=36827
--tserver_master_addrs=127.29.3.126:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:58.213366 13028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:58.214439 13028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:58.224114 13075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:58.224294 13077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:58.224133 13073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:58.224481 13028 server_base.cc:1034] running on GCE node
I20260811 22:20:58.224987 13028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:58.225847 13028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:58.227160 13028 hybrid_clock.cc:648] HybridClock initialized: now 1786486858227134 us; error 46 us; skew 500 ppm
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486858, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:58.230598 13028 init.cc:377] Logged in from keytab as oryx/127.29.3.65@KRBTEST.COM (short username oryx)
I20260811 22:20:58.231498 13028 webserver.cc:460] Webserver started at http://127.29.3.65:36827/ using document root <none> and password file <none>
I20260811 22:20:58.232038 13028 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:58.232111 13028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:58.236064 13028 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:58.238265 13087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.239616 13028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:20:58.239746 13028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "9a221ff2ff0241948219baecfe469507"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "561c92605e154a8db1ca0ab98e406119"
server_key_iv: "374dfc7b8c5d91555742429f4184da48"
server_key_version: "encryptionkey@0"
I20260811 22:20:58.240293 13028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:58.252158 13028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:58.255126 13028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:58.255297 13028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:58.255856 13028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:58.256701 13028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:58.256749 13028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.256801 13028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:58.256827 13028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.265419 13028 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:44853
I20260811 22:20:58.265470 13271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:44853 every 8 connection(s)
I20260811 22:20:58.266443 13028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:20:58.266866 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 13028
I20260811 22:20:58.267340 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:44679
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:40143
--webserver_port=36433
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.280439 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.269000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:37639)
0811 22:20:58.269496 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.269547 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.270805 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.272225 (+ 1420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.272232 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.272587 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.273048 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.273063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.274955 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.274959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.275810 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.275821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.276824 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.276827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.276956 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.277497 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.277515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.279709 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":120,"threads_started":1}
W20260811 22:20:58.280812 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.268751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:37639 (local address 127.29.3.126:40143)
0811 22:20:58.269349 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.269370 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.269680 (+ 310us) server_negotiation.cc:407] Connection header received
0811 22:20:58.271275 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.271290 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.271738 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.272134 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.273242 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.274819 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.275937 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.276680 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.276861 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.279971 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:37639: BlockingRecv error: recv got EOF from 127.29.3.65:37639 (error 108)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":120,"threads_started":1}
W20260811 22:20:58.280902 13272 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:40143 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.3.126:40143: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.285943 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.281344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:54507)
0811 22:20:58.281436 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.281450 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.281523 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.281736 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.281739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.281878 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.281963 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.281968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.282747 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.282750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.283286 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.283294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.283706 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.283709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.283795 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.284217 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.284243 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.285823 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260811 22:20:58.286348 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.281439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54507 (local address 127.29.3.126:40143)
0811 22:20:58.281552 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.281556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.281569 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:58.281604 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.281606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.281647 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.281711 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.282096 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.282650 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.283393 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.283612 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.283697 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.285965 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:54507: BlockingRecv error: recv got EOF from 127.29.3.65:54507 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.291635 13279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.286488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:60743)
0811 22:20:58.286590 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.286603 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.286682 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.287041 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.287043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.287158 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.287227 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.287231 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.288181 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.288184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.288700 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.288709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.289228 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.289230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.289283 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.289722 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.289733 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.291486 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 22:20:58.291790 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.286714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:60743 (local address 127.29.3.126:40143)
0811 22:20:58.286836 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.286840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.286852 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:58.286890 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.286893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.286937 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.287020 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.287315 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.288056 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.288807 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.289136 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.289251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.291674 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:60743: BlockingRecv error: recv got EOF from 127.29.3.65:60743 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 22:20:58.291867 13272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:20:58.384132 13276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:58.384382 13276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:58.384439 13276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:58.388404 13276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:58.388466 13276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:58.388545 13276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:20:58.393796 13276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.66.keytab
--principal=oryx/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:44679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=36433
--tserver_master_addrs=127.29.3.126:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:58.394870 13276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:58.395829 13276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:58.405493 13287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:58.405581 13276 server_base.cc:1034] running on GCE node
W20260811 22:20:58.405531 13295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:58.405824 13291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:58.406260 13276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:58.406941 13276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:58.408093 13276 hybrid_clock.cc:648] HybridClock initialized: now 1786486858408055 us; error 50 us; skew 500 ppm
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486858, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:58.410784 13276 init.cc:377] Logged in from keytab as oryx/127.29.3.66@KRBTEST.COM (short username oryx)
I20260811 22:20:58.411682 13276 webserver.cc:460] Webserver started at http://127.29.3.66:36433/ using document root <none> and password file <none>
I20260811 22:20:58.412230 13276 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:58.412312 13276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:58.416229 13276 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:20:58.418238 13316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.419476 13276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:58.419592 13276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "9d393d278e984b93b219e3eb63cd2528"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "730f7a06089de34a5d09702987748d7d"
server_key_iv: "99f918b0769aeaa591e2708ed5f41215"
server_key_version: "encryptionkey@0"
I20260811 22:20:58.420132 13276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:58.436620 13276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:58.439503 13276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:58.439692 13276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:58.440335 13276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:58.441223 13276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:58.441273 13276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.441327 13276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:58.441355 13276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.449963 13276 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:44679
I20260811 22:20:58.449987 13521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:44679 every 8 connection(s)
I20260811 22:20:58.451062 13276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:20:58.454948 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 13276
I20260811 22:20:58.455365 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:40035
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:40143
--webserver_port=42333
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
W20260811 22:20:58.461949 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.452904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:59227 (local address 127.29.3.126:40143)
0811 22:20:58.453037 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.453041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.453819 (+ 778us) server_negotiation.cc:407] Connection header received
0811 22:20:58.455158 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.455161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.455227 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.455303 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.456503 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.457038 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.458146 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.458416 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.458501 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.461821 (+ 3320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:59227: BlockingRecv error: recv got EOF from 127.29.3.66:59227 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:20:58.462611 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.453244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:59227)
0811 22:20:58.453667 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.453709 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.454927 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.455508 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.455518 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.455867 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.456354 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.456367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.457156 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.457158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.458024 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.458033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.458559 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.458563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.458728 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.459250 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.459265 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.461681 (+ 2416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
W20260811 22:20:58.463044 13523 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:40143 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.3.126:40143: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.468765 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.463524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:44179)
0811 22:20:58.463725 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.463739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.463830 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.464058 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.464061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.464226 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.464344 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.464352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.465054 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.465057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.465915 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.465928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.466343 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.466346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.466443 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.466872 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.466887 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.468634 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260811 22:20:58.468858 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.463636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:44179 (local address 127.29.3.126:40143)
0811 22:20:58.463741 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.463744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.463784 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:20:58.463915 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.463917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.463954 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.464020 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.464465 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.464951 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.466021 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.466224 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.466332 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.468755 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:44179: BlockingRecv error: recv got EOF from 127.29.3.66:44179 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.474406 13525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.469387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:39283)
0811 22:20:58.469535 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.469547 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.469621 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.469846 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.469849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.470001 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.470086 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.470091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.470771 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.470774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.471612 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.471622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.471983 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.471987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.472068 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.472571 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.472585 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.474273 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89}
W20260811 22:20:58.474633 13523 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:20:58.474854 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.469453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:39283 (local address 127.29.3.126:40143)
0811 22:20:58.469548 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.469551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.469584 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:20:58.469702 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.469704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.469748 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.469826 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.470172 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.470675 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.471712 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.471903 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.471967 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.474450 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:39283: BlockingRecv error: recv got EOF from 127.29.3.66:39283 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260811 22:20:58.570765 13527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:20:58.571029 13527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:20:58.571070 13527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:20:58.574996 13527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:20:58.575057 13527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:20:58.575114 13527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:20:58.580338 13527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/krb5kdc/oryx_127.29.3.67.keytab
--principal=oryx/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:40035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=42333
--tserver_master_addrs=127.29.3.126:40143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:20:58.581373 13527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:20:58.582376 13527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:20:58.591802 13536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:58.592011 13527 server_base.cc:1034] running on GCE node
W20260811 22:20:58.591859 13541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:20:58.591820 13538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:20:58.592546 13527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:20:58.593159 13527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:20:58.594332 13527 hybrid_clock.cc:648] HybridClock initialized: now 1786486858594303 us; error 55 us; skew 500 ppm
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486858, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:20:58.596916 13527 init.cc:377] Logged in from keytab as oryx/127.29.3.67@KRBTEST.COM (short username oryx)
I20260811 22:20:58.597757 13527 webserver.cc:460] Webserver started at http://127.29.3.67:42333/ using document root <none> and password file <none>
I20260811 22:20:58.598330 13527 fs_manager.cc:359] Metadata directory not provided
I20260811 22:20:58.598416 13527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:20:58.602346 13527 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:20:58.604391 13553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.605568 13527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:20:58.605680 13527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "71747473336b4def8d61949214ef5480"
format_stamp: "Formatted at 2026-08-11 22:20:57 on dist-test-slave-4tdj"
server_key: "7a03ac3b1a7a16ee19835f38b8b8322f"
server_key_iv: "7c9e378c18fe738939eccc3e74903344"
server_key_version: "encryptionkey@0"
I20260811 22:20:58.606251 13527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:20:58.628664 13527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:20:58.631485 13527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:20:58.631661 13527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:20:58.632270 13527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:20:58.633116 13527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:20:58.633172 13527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.633227 13527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:20:58.633252 13527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:20:58.641686 13527 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:40035
I20260811 22:20:58.641716 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:40035 every 8 connection(s)
I20260811 22:20:58.642714 13527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.652284 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 13527
W20260811 22:20:58.653002 13277 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.644508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43769 (local address 127.29.3.126:40143)
0811 22:20:58.644635 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.644638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.645511 (+ 873us) server_negotiation.cc:407] Connection header received
0811 22:20:58.646834 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.646837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.646888 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.646951 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.648047 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.648552 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.649596 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.649793 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.649856 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.652877 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:43769: BlockingRecv error: recv got EOF from 127.29.3.67:43769 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 22:20:58.653717 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.644913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:43769)
0811 22:20:58.645353 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.645397 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.646617 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.647101 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.647107 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.647449 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.647935 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.647947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.648637 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.648639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.649508 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.649516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.649895 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.649898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.650040 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.650570 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.650586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.652741 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
W20260811 22:20:58.654247 13758 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:40143 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.3.126:40143: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.659758 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.654708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:43461)
0811 22:20:58.654805 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.654818 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.654892 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.655222 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.655224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.655381 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.655484 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.655490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.656176 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.656178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.656851 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.656862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.657283 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.657287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.657377 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.657813 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.657827 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.659632 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260811 22:20:58.659876 13762 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.654779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43461 (local address 127.29.3.126:40143)
0811 22:20:58.655019 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.655024 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.655036 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:20:58.655070 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.655073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.655112 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.655185 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.655579 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.656076 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.656954 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.657173 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.657271 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.659752 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:43461: BlockingRecv error: recv got EOF from 127.29.3.67:43461 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260811 22:20:58.662022 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.653290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.29.3.126:40143)
0811 22:20:58.653394 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.653398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.653411 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:20:58.653478 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.653482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.653524 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.653607 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.654413 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.655065 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.656070 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.656331 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.656403 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.657788 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:20:58.657795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:20:58.657813 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:20:58.657843 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:20:58.659677 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:58.660451 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:58.660457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:58.660462 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:58.660585 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:20:58.660855 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:20:58.660858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:20:58.660859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:20:58.661217 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:20:58.661412 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 22:20:58.661696 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 22:20:58.661917 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:58 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:58.664758 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:58.660281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:34717)
0811 22:20:58.660377 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:58.660391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:58.660464 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:58.660663 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:58.660666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:58.660807 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:58.660870 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.660875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.661550 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.661553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.662118 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:58.662129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.662593 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.662596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:58.662671 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.663042 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:58.663052 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:58.664656 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 22:20:58.664855 13762 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:58.660346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:34717 (local address 127.29.3.126:40143)
0811 22:20:58.660477 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:58.660481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:58.660509 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:20:58.660541 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:58.660543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:58.660578 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:58.660635 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:58.660975 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.661454 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.662245 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:58.662496 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:58.662572 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:58.664762 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:34717: BlockingRecv error: recv got EOF from 127.29.3.67:34717 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260811 22:20:58.665132 13758 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 22:20:59 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:59 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:59.297545 13763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:59.292518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:54213)
0811 22:20:59.292817 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:59.292830 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:59.292928 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:59.293148 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:59.293150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:59.293331 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:59.293452 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:59.293458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.294146 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.294148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:59.294756 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:59.294765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.295154 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.295156 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:59.295223 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:59.295687 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:59.295698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:59.297397 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
W20260811 22:20:59.297680 13768 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:59.292603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54213 (local address 127.29.3.126:40143)
0811 22:20:59.292859 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:59.292863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:59.292879 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:20:59.293004 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:59.293006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:59.293049 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:59.293124 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:59.293537 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.294065 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.294854 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.295073 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.295144 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:59.297549 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:54213: BlockingRecv error: recv got EOF from 127.29.3.65:54213 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
Aug 11 22:20:59 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:59 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:59.480360 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:59.475246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:37377)
0811 22:20:59.475537 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:59.475550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:59.475649 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:59.475872 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:59.475874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:59.476050 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:59.476175 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:59.476180 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.476883 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.476885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:59.477463 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:59.477470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.477879 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.477881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:59.477946 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:59.478431 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:59.478443 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:59.480229 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
W20260811 22:20:59.480494 13768 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:59.475328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:37377 (local address 127.29.3.126:40143)
0811 22:20:59.475457 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:59.475460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:59.475610 (+ 150us) server_negotiation.cc:407] Connection header received
0811 22:20:59.475720 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:59.475722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:59.475767 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:59.475842 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:59.476274 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.476799 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.477562 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.477795 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.477871 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:59.480383 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:37377: BlockingRecv error: recv got EOF from 127.29.3.66:37377 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 11 22:20:59 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:20:59 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:20:59.671325 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:20:59.665792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:58379)
0811 22:20:59.666053 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0811 22:20:59.666067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:20:59.666182 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:20:59.666472 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:20:59.666474 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:20:59.666677 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:20:59.666781 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:59.666787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.667485 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.667487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:59.668086 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:20:59.668099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.668518 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.668521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:20:59.668602 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:59.669096 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:20:59.669106 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:20:59.671160 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":72,"threads_started":1}
W20260811 22:20:59.671468 13768 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:20:59.665903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:58379 (local address 127.29.3.126:40143)
0811 22:20:59.666076 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 22:20:59.666080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:20:59.666136 (+ 56us) server_negotiation.cc:407] Connection header received
0811 22:20:59.666320 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:20:59.666323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:20:59.666379 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:20:59.666447 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:20:59.666871 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.667387 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.668178 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:20:59.668386 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:20:59.668457 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:20:59.671342 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:58379: BlockingRecv error: recv got EOF from 127.29.3.67:58379 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 11 22:21:00 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:00 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:00.303833 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:00.298380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:54549)
0811 22:21:00.298685 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:00.298700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:00.298794 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:00.299077 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:00.299080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:00.299261 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:00.299366 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:00.299372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.300095 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.300098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:00.300685 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:00.300695 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.301155 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.301157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:00.301217 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:00.301702 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:00.301715 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:00.303657 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":75,"threads_started":1}
W20260811 22:21:00.303962 13773 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:00.298459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54549 (local address 127.29.3.126:40143)
0811 22:21:00.298735 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:00.298739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:00.298753 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:21:00.298874 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:00.298876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:00.298921 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:00.298995 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:00.299474 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.299993 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.300818 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.301061 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.301145 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:00.303824 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:54549: BlockingRecv error: recv got EOF from 127.29.3.65:54549 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
Aug 11 22:21:00 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:00 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:00.497114 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:00.481186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:44221)
0811 22:21:00.481515 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:00.481528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:00.481626 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:00.481825 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:00.481827 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:00.482001 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:00.482101 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:00.482106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.482847 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.482850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:00.483423 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:00.483432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.483821 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.483823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:00.483881 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:00.495092 (+ 11211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:00.495107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:00.496963 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
W20260811 22:21:00.497252 13773 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:00.481250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:44221 (local address 127.29.3.126:40143)
0811 22:21:00.481379 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:00.481383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:00.481591 (+ 208us) server_negotiation.cc:407] Connection header received
0811 22:21:00.481697 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:00.481699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:00.481747 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:00.481812 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:00.482188 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.482745 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.483517 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.483736 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.483811 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:00.497125 (+ 13314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:44221: BlockingRecv error: recv got EOF from 127.29.3.66:44221 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 11 22:21:00 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:00 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:00.677382 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:00.672179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:49751)
0811 22:21:00.672472 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:00.672489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:00.672589 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:00.672881 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:00.672883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:00.673064 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:00.673163 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:00.673168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.673886 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.673888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:00.674488 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:00.674496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.674903 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.674906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:00.674961 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:00.675472 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:00.675487 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:00.677233 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
W20260811 22:21:00.677520 13773 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:00.672387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:49751 (local address 127.29.3.126:40143)
0811 22:21:00.672524 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:00.672528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:00.672544 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:21:00.672718 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:00.672721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:00.672778 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:00.672852 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:00.673257 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.673794 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.674599 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:00.674821 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:00.674897 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:00.677406 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:49751: BlockingRecv error: recv got EOF from 127.29.3.67:49751 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 22:21:01 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:01 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:01.310570 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:01.304763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:52997)
0811 22:21:01.305039 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:01.305056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:01.305172 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:01.305553 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:01.305555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:01.305788 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:01.305943 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:01.305953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.306910 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.306913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:01.307599 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:01.307611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.308011 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.308013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:01.308070 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:01.308541 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:01.308552 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:01.310448 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
W20260811 22:21:01.310735 13780 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:01.304848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:52997 (local address 127.29.3.126:40143)
0811 22:21:01.305118 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:01.305121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:01.305136 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:21:01.305310 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:01.305313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:01.305377 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:01.305495 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:01.306072 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.306792 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.307701 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.307924 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.308004 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:01.310589 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:52997: BlockingRecv error: recv got EOF from 127.29.3.65:52997 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":94,"threads_started":1}
Aug 11 22:21:01 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:01 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:01.503473 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:01.497953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:56111)
0811 22:21:01.498243 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:01.498257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:01.498389 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:01.498613 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:01.498615 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:01.498814 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:01.498918 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:01.498923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.499596 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.499599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:01.500183 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:01.500191 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.500692 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.500694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:01.500760 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:01.501233 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:01.501243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:01.503353 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
W20260811 22:21:01.503592 13780 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:01.498047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:56111 (local address 127.29.3.126:40143)
0811 22:21:01.498187 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:01.498190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:01.498335 (+ 145us) server_negotiation.cc:407] Connection header received
0811 22:21:01.498467 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:01.498469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:01.498517 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:01.498591 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:01.499007 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.499516 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.500337 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.500572 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.500660 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:01.503487 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:56111: BlockingRecv error: recv got EOF from 127.29.3.66:56111 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 11 22:21:01 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:01 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:01.683565 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:01.678177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:33447)
0811 22:21:01.678505 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:01.678518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:01.678631 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:01.678892 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:01.678895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:01.679098 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:01.679196 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:01.679202 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.679981 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.679985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:01.680557 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:01.680566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.681020 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.681024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:01.681105 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:01.681585 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:01.681596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:01.683376 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":145,"threads_started":1}
W20260811 22:21:01.683634 13780 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:01.678235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:33447 (local address 127.29.3.126:40143)
0811 22:21:01.678394 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:01.678398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:01.678590 (+ 192us) server_negotiation.cc:407] Connection header received
0811 22:21:01.678730 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:01.678733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:01.678783 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:01.678850 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:01.679331 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.679868 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.680677 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:01.680913 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:01.681003 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:01.683529 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:33447: BlockingRecv error: recv got EOF from 127.29.3.67:33447 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 11 22:21:02 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:02 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:02.316972 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:02.311344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:54509)
0811 22:21:02.311640 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:02.311654 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:02.311752 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:02.311980 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:02.311981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:02.312174 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:02.312275 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:02.312280 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.313032 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.313035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:02.313615 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:02.313625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.314049 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.314051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:02.314111 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:02.314669 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:02.314686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:02.316855 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
W20260811 22:21:02.317109 13789 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:02.311462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54509 (local address 127.29.3.126:40143)
0811 22:21:02.311696 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:02.311700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:02.311730 (+ 30us) server_negotiation.cc:407] Connection header received
0811 22:21:02.311832 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:02.311834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:02.311880 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:02.311956 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:02.312366 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.312902 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.313728 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.313954 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.314031 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:02.316993 (+ 2962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:54509: BlockingRecv error: recv got EOF from 127.29.3.65:54509 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":74,"threads_started":1}
Aug 11 22:21:02 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:02 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:02.509513 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:02.504227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:48919)
0811 22:21:02.504492 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:02.504506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:02.504606 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:02.504833 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:02.504835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:02.505010 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:02.505111 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:02.505117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.505814 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.505816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:02.506553 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:02.506565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.506996 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.507000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:02.507087 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:02.507592 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:02.507604 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:02.509383 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":96,"threads_started":1}
W20260811 22:21:02.509684 13789 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:02.504301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:48919 (local address 127.29.3.126:40143)
0811 22:21:02.504467 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:02.504472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:02.504568 (+ 96us) server_negotiation.cc:407] Connection header received
0811 22:21:02.504679 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:02.504681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:02.504741 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:02.504808 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:02.505195 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.505729 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.506670 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.506902 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.506982 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:02.509542 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:48919: BlockingRecv error: recv got EOF from 127.29.3.66:48919 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 11 22:21:02 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:02 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:02.689663 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:02.684398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:41909)
0811 22:21:02.684692 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:02.684704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:02.684843 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:02.685083 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:02.685086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:02.685281 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:02.685402 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:02.685408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.686143 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.686145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:02.686771 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:02.686782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.687257 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.687260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:02.687327 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:02.687790 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:02.687801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:02.689543 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
W20260811 22:21:02.689836 13789 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:02.684461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:41909 (local address 127.29.3.126:40143)
0811 22:21:02.684603 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:02.684608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:02.684798 (+ 190us) server_negotiation.cc:407] Connection header received
0811 22:21:02.684937 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:02.684939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:02.684991 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:02.685053 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:02.685501 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.686058 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.686895 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:02.687145 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:02.687237 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:02.689703 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:41909: BlockingRecv error: recv got EOF from 127.29.3.67:41909 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 11 22:21:03 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:03 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:03.323448 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:03.317866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:54231)
0811 22:21:03.318176 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:03.318189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:03.318313 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:03.318548 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:03.318551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:03.318735 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:03.318834 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:03.318839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.319621 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.319624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:03.320222 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:03.320232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.320660 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.320663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:03.320723 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:03.321282 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:03.321294 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:03.323311 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
W20260811 22:21:03.323587 13797 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:03.317943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54231 (local address 127.29.3.126:40143)
0811 22:21:03.318178 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:03.318182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:03.318256 (+ 74us) server_negotiation.cc:407] Connection header received
0811 22:21:03.318389 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:03.318391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:03.318437 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:03.318520 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:03.318922 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.319468 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.320335 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.320562 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.320636 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:03.323452 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:54231: BlockingRecv error: recv got EOF from 127.29.3.65:54231 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
Aug 11 22:21:03 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:03 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:03.515560 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:03.510312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:52203)
0811 22:21:03.510564 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:03.510578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:03.510672 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:03.510877 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:03.510879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:03.511057 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:03.511158 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:03.511163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.511890 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.511892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:03.512469 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:03.512476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.512837 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.512839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:03.512899 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:03.513357 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:03.513368 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:03.515395 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":72,"threads_started":1}
W20260811 22:21:03.515626 13797 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:03.510434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:52203 (local address 127.29.3.126:40143)
0811 22:21:03.510589 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:03.510592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:03.510628 (+ 36us) server_negotiation.cc:407] Connection header received
0811 22:21:03.510742 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:03.510744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:03.510787 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:03.510853 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:03.511281 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.511802 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.512552 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.512760 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.512831 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:03.515526 (+ 2695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:52203: BlockingRecv error: recv got EOF from 127.29.3.66:52203 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 11 22:21:03 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:03 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:03.696206 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:03.690763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:40977)
0811 22:21:03.691064 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:03.691077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:03.691202 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:03.691575 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:03.691577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:03.691785 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:03.691900 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:03.691908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.692755 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.692758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:03.693344 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:03.693353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.693772 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.693774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:03.693840 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:03.694271 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:03.694282 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:03.696084 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":89,"threads_started":1}
W20260811 22:21:03.696372 13797 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:03.690780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:40977 (local address 127.29.3.126:40143)
0811 22:21:03.690926 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:03.690931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:03.691176 (+ 245us) server_negotiation.cc:407] Connection header received
0811 22:21:03.691298 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:03.691301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:03.691357 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:03.691445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:03.692011 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.692654 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.693449 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:03.693682 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:03.693763 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:03.696257 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:40977: BlockingRecv error: recv got EOF from 127.29.3.67:40977 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 22:21:04 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:04 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:04.330185 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:04.324362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:60749)
0811 22:21:04.324680 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:04.324696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:04.324806 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:04.325056 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:04.325059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:04.325276 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:04.325387 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:04.325393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.326131 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.326134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:04.326778 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:04.326791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.327224 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.327230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:04.327317 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:04.327916 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:04.327930 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:04.330066 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":90,"threads_started":1}
W20260811 22:21:04.330359 13803 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:04.324476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:60749 (local address 127.29.3.126:40143)
0811 22:21:04.324748 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:04.324751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:04.324765 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:21:04.324898 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:04.324900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:04.324944 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:04.325020 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:04.325484 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.326026 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.326877 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.327093 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.327166 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:04.330210 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:60749: BlockingRecv error: recv got EOF from 127.29.3.65:60749 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":61,"threads_started":1}
Aug 11 22:21:04 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:04 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:04.521446 13804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:04.516345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:60637)
0811 22:21:04.516628 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:04.516642 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:04.516757 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:04.516972 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:04.516974 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:04.517158 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:04.517258 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:04.517264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.517932 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.517934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:04.518548 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:04.518559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.518919 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.518921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:04.518987 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:04.519484 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:04.519495 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:04.521291 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
W20260811 22:21:04.521571 13803 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:04.516426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:60637 (local address 127.29.3.126:40143)
0811 22:21:04.516566 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:04.516570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:04.516715 (+ 145us) server_negotiation.cc:407] Connection header received
0811 22:21:04.516829 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:04.516831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:04.516876 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:04.516948 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:04.517343 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.517853 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.518638 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.518843 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.518911 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:04.521441 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:60637: BlockingRecv error: recv got EOF from 127.29.3.66:60637 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 22:21:04 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:04 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:04.701875 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:04.696926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:33585)
0811 22:21:04.697197 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:04.697211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:04.697316 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:04.697524 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:04.697526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:04.697709 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:04.697826 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:04.697831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.698534 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.698536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:04.699090 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:04.699098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.699468 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.699470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:04.699529 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:04.699984 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:04.699995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:04.701720 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":95,"threads_started":1}
W20260811 22:21:04.701969 13803 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:04.697008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:33585 (local address 127.29.3.126:40143)
0811 22:21:04.697139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:04.697143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:04.697272 (+ 129us) server_negotiation.cc:407] Connection header received
0811 22:21:04.697384 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:04.697386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:04.697432 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:04.697495 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:04.697909 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.698453 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.699186 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:04.699394 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:04.699458 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:04.701866 (+ 2408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:33585: BlockingRecv error: recv got EOF from 127.29.3.67:33585 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 11 22:21:05 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:05 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:05.336483 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:05.331068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:51257)
0811 22:21:05.331524 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:05.331537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:05.331646 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:05.331909 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:05.331912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:05.332122 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:05.332224 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:05.332231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.332974 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.332978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:05.333532 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:05.333541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.333983 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.333987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:05.334067 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:05.334552 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:05.334564 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:05.336360 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":73,"threads_started":1}
W20260811 22:21:05.336689 13809 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:05.331149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:51257 (local address 127.29.3.126:40143)
0811 22:21:05.331419 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:05.331424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:05.331608 (+ 184us) server_negotiation.cc:407] Connection header received
0811 22:21:05.331738 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:05.331742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:05.331794 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:05.331870 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:05.332343 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.332870 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.333647 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.333878 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.333967 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:05.336549 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:51257: BlockingRecv error: recv got EOF from 127.29.3.65:51257 (error 108)
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":90,"threads_started":1}
Aug 11 22:21:05 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:05 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:05.531955 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:05.522371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:56815)
0811 22:21:05.522723 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:05.522736 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:05.522851 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:05.523106 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:05.523109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:05.523318 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:05.523411 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:05.523417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.524162 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.524165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:05.524726 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:05.524734 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.525174 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.525177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:05.525278 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:05.529931 (+ 4653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:05.529945 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:05.531798 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":82,"threads_started":1}
W20260811 22:21:05.532079 13809 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:05.522483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:56815 (local address 127.29.3.126:40143)
0811 22:21:05.522645 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:05.522651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:05.522808 (+ 157us) server_negotiation.cc:407] Connection header received
0811 22:21:05.522943 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:05.522946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:05.523000 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:05.523081 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:05.523526 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.524073 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.524835 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.525073 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.525156 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:05.531974 (+ 6818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:56815: BlockingRecv error: recv got EOF from 127.29.3.66:56815 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 11 22:21:05 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:05 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:05.708367 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:05.702680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:39673)
0811 22:21:05.703047 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:05.703061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:05.703173 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:05.703416 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:05.703419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:05.703626 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:05.703731 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:05.703737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.704489 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.704492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:05.705056 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:05.705064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.705554 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.705558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:05.705645 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:05.706147 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:05.706160 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:05.708211 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":107,"threads_started":1}
W20260811 22:21:05.708469 13809 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:05.702782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:39673 (local address 127.29.3.126:40143)
0811 22:21:05.702927 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:05.702932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:05.703133 (+ 201us) server_negotiation.cc:407] Connection header received
0811 22:21:05.703262 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:05.703265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:05.703317 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:05.703386 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:05.703845 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.704381 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.705177 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:05.705413 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:05.705542 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:05.708365 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:39673: BlockingRecv error: recv got EOF from 127.29.3.67:39673 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 22:21:06 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:06 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:06.342793 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:06.337442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:38145)
0811 22:21:06.337758 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:06.337772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:06.337868 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:06.338142 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:06.338146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:06.338411 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:06.338543 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:06.338552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.339423 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.339425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:06.339986 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:06.339995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.340367 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.340369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:06.340426 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:06.340886 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:06.340898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:06.342637 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
W20260811 22:21:06.342892 13816 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:06.337656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38145 (local address 127.29.3.126:40143)
0811 22:21:06.337898 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:06.337902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:06.337914 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:21:06.337951 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:06.337954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:06.338003 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:06.338104 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:06.338661 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.339321 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.340080 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.340291 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.340356 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:06.342778 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:38145: BlockingRecv error: recv got EOF from 127.29.3.65:38145 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
Aug 11 22:21:06 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:06 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:06.538691 13817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:06.532874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:56629)
0811 22:21:06.533166 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:06.533182 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:06.533298 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:06.533563 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:06.533566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:06.533789 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:06.533909 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:06.533917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.535061 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.535064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:06.535625 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:06.535634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.536005 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.536007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:06.536067 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:06.536589 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:06.536603 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:06.538538 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":94,"threads_started":1}
W20260811 22:21:06.538801 13816 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:06.532979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:56629 (local address 127.29.3.126:40143)
0811 22:21:06.533136 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:06.533141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:06.533251 (+ 110us) server_negotiation.cc:407] Connection header received
0811 22:21:06.533390 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:06.533394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:06.533438 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:06.533511 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:06.534040 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.534960 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.535714 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.535927 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.535997 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:06.538700 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:56629: BlockingRecv error: recv got EOF from 127.29.3.66:56629 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 22:21:06 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:06 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:06.714532 13818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:06.709118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:47031)
0811 22:21:06.709419 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:06.709434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:06.709548 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:06.709802 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:06.709805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:06.710005 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:06.710102 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:06.710108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.710971 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.710975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:06.711542 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:06.711551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.711981 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.711985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:06.712066 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:06.712553 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:06.712565 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:06.714381 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":108,"threads_started":1}
W20260811 22:21:06.714636 13816 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:06.709224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:47031 (local address 127.29.3.126:40143)
0811 22:21:06.709357 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:06.709361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:06.709503 (+ 142us) server_negotiation.cc:407] Connection header received
0811 22:21:06.709633 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:06.709637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:06.709702 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:06.709768 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:06.710217 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.710858 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.711663 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:06.711882 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:06.711968 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:06.714534 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:47031: BlockingRecv error: recv got EOF from 127.29.3.67:47031 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 11 22:21:07 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:07 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:07.348652 13819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:07.343613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:59025)
0811 22:21:07.343894 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:07.343908 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:07.343999 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:07.344212 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:07.344214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:07.344408 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:07.344505 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:07.344510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.345289 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.345291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:07.345846 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:07.345854 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.346221 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.346223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:07.346280 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:07.346762 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:07.346772 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:07.348534 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
W20260811 22:21:07.348819 13822 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:07.343678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:59025 (local address 127.29.3.126:40143)
0811 22:21:07.343938 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:07.343941 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:07.343954 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:21:07.344074 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:07.344077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:07.344123 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:07.344201 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:07.344595 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.345178 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.345932 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.346144 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.346216 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:07.348669 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:59025: BlockingRecv error: recv got EOF from 127.29.3.65:59025 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":76,"threads_started":1}
Aug 11 22:21:07 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:07 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:07.544415 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:07.539402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:51483)
0811 22:21:07.539670 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:07.539683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:07.539782 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:07.539986 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:07.539988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:07.540165 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:07.540263 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:07.540268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.540982 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.540985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:07.541553 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:07.541561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.541924 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.541926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:07.541991 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:07.542482 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:07.542493 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:07.544249 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
W20260811 22:21:07.544682 13822 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:07.539518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:51483 (local address 127.29.3.126:40143)
0811 22:21:07.539677 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:07.539680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:07.539745 (+ 65us) server_negotiation.cc:407] Connection header received
0811 22:21:07.539852 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:07.539854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:07.539897 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:07.539962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:07.540363 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.540883 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.541636 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.541848 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.541915 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:07.544439 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:51483: BlockingRecv error: recv got EOF from 127.29.3.66:51483 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 11 22:21:07 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:07 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:07.720952 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:07.715420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:53675)
0811 22:21:07.715687 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:07.715701 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:07.715821 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:07.716078 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:07.716081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:07.716263 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:07.716366 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:07.716371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.717097 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.717099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:07.717648 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:07.717655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.718056 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.718058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:07.718123 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:07.718662 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:07.718674 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:07.720831 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":80,"threads_started":1}
W20260811 22:21:07.721125 13822 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:07.715483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:53675 (local address 127.29.3.126:40143)
0811 22:21:07.715603 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:07.715606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:07.715776 (+ 170us) server_negotiation.cc:407] Connection header received
0811 22:21:07.715914 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:07.715916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:07.715966 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:07.716050 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:07.716462 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.717015 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.717744 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:07.717969 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:07.718045 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:07.720995 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:53675: BlockingRecv error: recv got EOF from 127.29.3.67:53675 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 22:21:08 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:08 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:08.354627 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:08.349421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:34675)
0811 22:21:08.349747 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:08.349759 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:08.349858 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:08.350082 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:08.350084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:08.350271 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:08.350400 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:08.350406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.351113 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.351115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:08.351676 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:08.351683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.352109 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.352111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:08.352169 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:08.352695 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:08.352710 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:08.354499 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":126,"threads_started":1}
W20260811 22:21:08.354776 13831 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:08.349524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:34675 (local address 127.29.3.126:40143)
0811 22:21:08.349755 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:08.349759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:08.349825 (+ 66us) server_negotiation.cc:407] Connection header received
0811 22:21:08.349926 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:08.349929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:08.349975 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:08.350060 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:08.350492 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.351028 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.351784 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.352019 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.352103 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:08.354639 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:34675: BlockingRecv error: recv got EOF from 127.29.3.65:34675 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":60,"threads_started":1}
Aug 11 22:21:08 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:08 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:08.550264 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:08.545185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:41213)
0811 22:21:08.545485 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:08.545500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:08.545598 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:08.545822 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:08.545824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:08.546004 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:08.546107 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:08.546113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.546832 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.546835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:08.547392 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:08.547399 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.547777 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.547779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:08.547837 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:08.548307 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:08.548318 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:08.550098 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
W20260811 22:21:08.550529 13831 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:08.545265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:41213 (local address 127.29.3.126:40143)
0811 22:21:08.545400 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:08.545403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:08.545563 (+ 160us) server_negotiation.cc:407] Connection header received
0811 22:21:08.545676 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:08.545678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:08.545723 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:08.545796 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:08.546195 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.546753 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.547479 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.547698 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.547764 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:08.550254 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:41213: BlockingRecv error: recv got EOF from 127.29.3.66:41213 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Aug 11 22:21:08 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:08 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:08.727439 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:08.721796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:36545)
0811 22:21:08.722089 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:08.722104 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:08.722192 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:08.722439 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:08.722441 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:08.722624 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:08.722731 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:08.722738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.723502 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.723506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:08.724150 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:08.724161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.724562 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.724565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:08.724634 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:08.725199 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:08.725214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:08.727280 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":87,"threads_started":1}
W20260811 22:21:08.727617 13831 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:08.721980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:36545 (local address 127.29.3.126:40143)
0811 22:21:08.722125 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:08.722129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:08.722147 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:21:08.722275 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:08.722277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:08.722346 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:08.722413 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:08.722829 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.723384 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.724257 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:08.724475 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:08.724549 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:08.727487 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:36545: BlockingRecv error: recv got EOF from 127.29.3.67:36545 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 11 22:21:09 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:09 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:09.360285 13837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:09.355374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:47587)
0811 22:21:09.355663 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:09.355677 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:09.355770 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:09.355990 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:09.355992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:09.356180 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:09.356280 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:09.356285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.356976 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.356978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:09.357532 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:09.357541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.357901 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.357904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:09.357961 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:09.358442 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:09.358454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:09.360170 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
W20260811 22:21:09.360435 13840 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:09.355456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:47587 (local address 127.29.3.126:40143)
0811 22:21:09.355708 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:09.355720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:09.355732 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:21:09.355848 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:09.355850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:09.355894 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:09.355966 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:09.356366 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.356893 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.357614 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.357827 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.357893 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:09.360310 (+ 2417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:47587: BlockingRecv error: recv got EOF from 127.29.3.65:47587 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
Aug 11 22:21:09 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:09 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:09.556442 13841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:09.551120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:55809)
0811 22:21:09.551398 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:09.551411 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:09.551511 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:09.551738 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:09.551740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:09.551916 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:09.552017 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:09.552022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.552694 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.552697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:09.553256 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:09.553263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.553615 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.553618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:09.553674 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:09.554164 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:09.554176 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:09.556292 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":94,"threads_started":1}
W20260811 22:21:09.556542 13840 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:09.551231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:55809 (local address 127.29.3.126:40143)
0811 22:21:09.551370 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:09.551374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:09.551471 (+ 97us) server_negotiation.cc:407] Connection header received
0811 22:21:09.551580 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:09.551582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:09.551627 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:09.551713 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:09.552100 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.552604 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.553335 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.553541 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.553606 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:09.556440 (+ 2834us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:55809: BlockingRecv error: recv got EOF from 127.29.3.66:55809 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 22:21:09 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:09 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:09.733490 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:09.728292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:43619)
0811 22:21:09.728587 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:09.728600 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:09.728719 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:09.728958 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:09.728961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:09.729164 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:09.729264 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:09.729270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.730015 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.730018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:09.730597 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:09.730606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.731047 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.731051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:09.731133 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:09.731582 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:09.731593 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:09.733311 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":116,"threads_started":1}
W20260811 22:21:09.733569 13840 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:09.728369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43619 (local address 127.29.3.126:40143)
0811 22:21:09.728497 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:09.728501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:09.728667 (+ 166us) server_negotiation.cc:407] Connection header received
0811 22:21:09.728803 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:09.728806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:09.728854 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:09.728919 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:09.729385 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.729909 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.730720 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:09.730945 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:09.731031 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:09.733465 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:43619: BlockingRecv error: recv got EOF from 127.29.3.67:43619 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 22:21:10 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:10 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:10.366036 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:10.361002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:55953)
0811 22:21:10.361314 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:10.361327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:10.361423 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:10.361661 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:10.361664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:10.361849 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:10.361949 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:10.361954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.362709 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.362712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:10.363265 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:10.363272 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.363628 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.363630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:10.363689 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:10.364125 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:10.364135 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:10.365917 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":120,"threads_started":1}
W20260811 22:21:10.366225 13845 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:10.361081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:55953 (local address 127.29.3.126:40143)
0811 22:21:10.361337 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:10.361342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:10.361380 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:21:10.361502 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:10.361505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:10.361553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:10.361631 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:10.362067 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.362632 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.363346 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.363553 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.363621 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:10.366083 (+ 2462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:55953: BlockingRecv error: recv got EOF from 127.29.3.65:55953 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
Aug 11 22:21:10 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:10 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:10.566805 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:10.557246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:60629)
0811 22:21:10.557578 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:10.557593 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:10.557697 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:10.557974 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:10.557977 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:10.558162 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:10.558278 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:10.558283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.559057 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.559059 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:10.559642 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:10.559649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.560089 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.560092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:10.560164 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:10.564826 (+ 4662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:10.564839 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:10.566655 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":110,"threads_started":1}
W20260811 22:21:10.567059 13845 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:10.557319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:60629 (local address 127.29.3.126:40143)
0811 22:21:10.557491 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:10.557496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:10.557687 (+ 191us) server_negotiation.cc:407] Connection header received
0811 22:21:10.557779 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:10.557781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:10.557839 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:10.557902 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:10.558416 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.558967 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.559754 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.559980 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.560060 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:10.566822 (+ 6762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:60629: BlockingRecv error: recv got EOF from 127.29.3.66:60629 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 11 22:21:10 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:10 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:10.739451 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:10.734370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:53251)
0811 22:21:10.734636 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:10.734651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:10.734770 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:10.734970 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:10.734971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:10.735147 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:10.735248 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:10.735253 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.735974 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.735977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:10.736544 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:10.736553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.736990 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.736992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:10.737051 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:10.737503 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:10.737513 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:10.739330 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":72,"threads_started":1}
W20260811 22:21:10.739578 13845 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:10.734453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:53251 (local address 127.29.3.126:40143)
0811 22:21:10.734597 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:10.734601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:10.734712 (+ 111us) server_negotiation.cc:407] Connection header received
0811 22:21:10.734843 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:10.734845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:10.734893 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:10.734957 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:10.735330 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.735869 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.736672 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:10.736902 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:10.736984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:10.739469 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:53251: BlockingRecv error: recv got EOF from 127.29.3.67:53251 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 22:21:11 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:11 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:11.372345 13854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:11.366866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:46567)
0811 22:21:11.367175 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:11.367189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:11.367284 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:11.367513 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:11.367515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:11.367702 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:11.367799 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:11.367804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.368495 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.368497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:11.369056 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:11.369065 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.369485 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.369488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:11.369639 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:11.370128 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:11.370139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:11.372173 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":77,"threads_started":1}
W20260811 22:21:11.372440 13855 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:11.366975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:46567 (local address 127.29.3.126:40143)
0811 22:21:11.367212 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:11.367216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:11.367233 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:21:11.367361 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:11.367363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:11.367406 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:11.367487 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:11.367887 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.368412 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.369144 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.369361 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.369434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:11.372321 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:46567: BlockingRecv error: recv got EOF from 127.29.3.65:46567 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":71,"threads_started":1}
Aug 11 22:21:11 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:11 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:11.572424 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:11.567572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:54797)
0811 22:21:11.567825 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:11.567838 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:11.567935 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:11.568147 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:11.568149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:11.568333 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:11.568436 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:11.568441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.569117 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.569119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:11.569683 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:11.569690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.570045 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.570047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:11.570106 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:11.570575 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:11.570586 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:11.572294 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":86,"threads_started":1}
W20260811 22:21:11.572541 13855 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:11.567678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:54797 (local address 127.29.3.126:40143)
0811 22:21:11.567833 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:11.567836 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:11.567897 (+ 61us) server_negotiation.cc:407] Connection header received
0811 22:21:11.568008 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:11.568011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:11.568055 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:11.568124 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:11.568520 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.569035 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.569764 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.569971 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.570038 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:11.572424 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:54797: BlockingRecv error: recv got EOF from 127.29.3.66:54797 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 11 22:21:11 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:11 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:11.745975 13858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:11.740226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:49117)
0811 22:21:11.740531 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:11.740546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:11.740653 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:11.740873 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:11.740875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:11.741095 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:11.741210 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:11.741216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.741978 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.741982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:11.742612 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:11.742622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.743044 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.743046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:11.743106 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:11.743624 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:11.743636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:11.745800 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":101,"threads_started":1}
W20260811 22:21:11.746125 13855 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:11.740318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:49117 (local address 127.29.3.126:40143)
0811 22:21:11.740462 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:11.740467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:11.740640 (+ 173us) server_negotiation.cc:407] Connection header received
0811 22:21:11.740726 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:11.740728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:11.740778 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:11.740850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:11.741317 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.741862 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.742725 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:11.742957 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:11.743035 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:11.745999 (+ 2964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:49117: BlockingRecv error: recv got EOF from 127.29.3.67:49117 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 22:21:12 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:12 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:12.378608 13860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:12.373142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:36273)
0811 22:21:12.373415 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:12.373429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:12.373522 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:12.373820 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:12.373823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:12.374040 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:12.374182 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:12.374192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.375171 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.375174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:12.375735 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:12.375744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.376109 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.376111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:12.376170 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:12.376679 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:12.376693 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:12.378441 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
W20260811 22:21:12.378712 13862 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:12.373217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:36273 (local address 127.29.3.126:40143)
0811 22:21:12.373508 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:12.373512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:12.373525 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:21:12.373614 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:12.373617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:12.373674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:12.373771 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:12.374327 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.375068 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.375823 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.376035 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.376102 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:12.378572 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:36273: BlockingRecv error: recv got EOF from 127.29.3.65:36273 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":67,"threads_started":1}
Aug 11 22:21:12 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:12 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:12.578444 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:12.573190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:42967)
0811 22:21:12.573491 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:12.573513 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:12.573610 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:12.573941 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:12.573944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:12.574133 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:12.574243 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:12.574249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.574992 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.574994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:12.575555 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:12.575562 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.575961 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.575964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:12.576031 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:12.576490 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:12.576503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:12.578323 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":110,"threads_started":1}
W20260811 22:21:12.578593 13862 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:12.573288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:42967 (local address 127.29.3.126:40143)
0811 22:21:12.573433 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:12.573437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:12.573571 (+ 134us) server_negotiation.cc:407] Connection header received
0811 22:21:12.573693 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:12.573695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:12.573741 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:12.573799 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:12.574378 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.574908 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.575648 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.575866 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.575932 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:12.578468 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:42967: BlockingRecv error: recv got EOF from 127.29.3.66:42967 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 22:21:12 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:12 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.67@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:12.753729 13867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:12.747584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.67:40703)
0811 22:21:12.748011 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:12.748023 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:12.748135 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:12.748435 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:12.748439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:12.748694 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:12.748828 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:12.748836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.749848 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.749852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:12.750665 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:12.750677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.751148 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.751151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:12.751235 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:12.751725 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:12.751740 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:12.753549 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":147,"threads_started":1}
W20260811 22:21:12.753808 13862 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:12.747651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:40703 (local address 127.29.3.126:40143)
0811 22:21:12.747793 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:12.747798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:12.748098 (+ 300us) server_negotiation.cc:407] Connection header received
0811 22:21:12.748239 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:12.748242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:12.748295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:12.748367 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:12.748970 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.749715 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.750789 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:12.751035 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:12.751129 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:12.753689 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.67:40703: BlockingRecv error: recv got EOF from 127.29.3.67:40703 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 11 22:21:13 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:13 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.65@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:13.384840 13868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:13.379601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.65:58071)
0811 22:21:13.379866 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:13.379879 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:13.379965 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:13.380203 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:13.380206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:13.380418 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:13.380517 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:13.380524 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.381228 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.381230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:13.381784 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:13.381792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.382149 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.382151 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:13.382205 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:13.382755 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:13.382767 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:13.384708 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
W20260811 22:21:13.384986 13870 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:13.379714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:58071 (local address 127.29.3.126:40143)
0811 22:21:13.379953 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:13.379957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:13.379970 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:21:13.380046 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:13.380048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:13.380092 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:13.380200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:13.380616 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.381136 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.381866 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.382074 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.382140 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:13.384850 (+ 2710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.65:58071: BlockingRecv error: recv got EOF from 127.29.3.65:58071 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
Aug 11 22:21:13 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
Aug 11 22:21:13 dist-test-slave-4tdj krb5kdc[12012](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.3.66@KRBTEST.COM for oryx/127.29.3.126@KRBTEST.COM, Server not found in Kerberos database
I20260811 22:21:13.584532 13871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:13.579320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:40143 (local address 127.29.3.66:59393)
0811 22:21:13.579600 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:13.579614 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:13.579705 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:13.579973 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:13.579975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:13.580168 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:13.580273 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:13.580279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.581101 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.581105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:13.581673 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:13.581682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.582052 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.582054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:13.582127 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:13.582631 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:13.582643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:13.584408 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.3.126:40143: Server oryx/127.29.3.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
W20260811 22:21:13.584649 13870 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:13.579402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:59393 (local address 127.29.3.126:40143)
0811 22:21:13.579537 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:13.579541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:13.579672 (+ 131us) server_negotiation.cc:407] Connection header received
0811 22:21:13.579777 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:13.579779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:13.579825 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:13.579892 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:13.580369 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.580971 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.581765 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.581974 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.582042 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:13.584550 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.3.66:59393: BlockingRecv error: recv got EOF from 127.29.3.66:59393 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:21:13.659227 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 13028
I20260811 22:21:13.673079 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 13276
I20260811 22:21:13.684413 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 13527
I20260811 22:21:13.704413 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 12910
2026-08-11T22:21:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18275 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T22:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:21:13Z Disabled control of system clock
I20260811 22:21:13.744769 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36329
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:36329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 22:21:13.859165 13876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:13.859462 13876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:13.859504 13876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:13.863677 13876 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:21:13.863749 13876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:13.863763 13876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:13.863795 13876 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:21:13.863826 13876 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:21:13.868757 13876 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:36329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:13.869892 13876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:13.871395 13876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:13.880108 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:13.880155 13887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:13.880108 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:13.880297 13876 server_base.cc:1034] running on GCE node
I20260811 22:21:13.881094 13876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:13.882385 13876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:13.883563 13876 hybrid_clock.cc:648] HybridClock initialized: now 1786486873883546 us; error 38 us; skew 500 ppm
I20260811 22:21:13.885581 13876 webserver.cc:460] Webserver started at http://127.29.3.126:41317/ using document root <none> and password file <none>
I20260811 22:21:13.886339 13876 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:13.886436 13876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:13.886709 13876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:13.888934 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "68df8d3bf5ef44f29e8770d738f940d4"
format_stamp: "Formatted at 2026-08-11 22:21:13 on dist-test-slave-4tdj"
server_key: "28f3897609232bc7a4a4d6054b4a5db3"
server_key_iv: "2fccf08eef32451a3250ec8492db6f2c"
server_key_version: "encryptionkey@0"
I20260811 22:21:13.889555 13876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "68df8d3bf5ef44f29e8770d738f940d4"
format_stamp: "Formatted at 2026-08-11 22:21:13 on dist-test-slave-4tdj"
server_key: "28f3897609232bc7a4a4d6054b4a5db3"
server_key_iv: "2fccf08eef32451a3250ec8492db6f2c"
server_key_version: "encryptionkey@0"
I20260811 22:21:13.893410 13876 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260811 22:21:13.896041 13895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:13.897349 13876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:21:13.897480 13876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "68df8d3bf5ef44f29e8770d738f940d4"
format_stamp: "Formatted at 2026-08-11 22:21:13 on dist-test-slave-4tdj"
server_key: "28f3897609232bc7a4a4d6054b4a5db3"
server_key_iv: "2fccf08eef32451a3250ec8492db6f2c"
server_key_version: "encryptionkey@0"
I20260811 22:21:13.897620 13876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:13.920365 13876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:13.921144 13876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:13.921327 13876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:13.928408 13876 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:36329
I20260811 22:21:13.928436 13998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:36329 every 8 connection(s)
I20260811 22:21:13.929397 13876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:21:13.931196 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 13876
I20260811 22:21:13.931308 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:21:13.931566 29709 external_mini_cluster.cc:1428] Setting key 02d9a35c230901ed8e8efc2f61607799
I20260811 22:21:13.933058 14000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:21:13.938941 14000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4: Bootstrap starting.
I20260811 22:21:13.940685 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:13.932936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.29.3.126:36329)
0811 22:21:13.933618 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:13.933635 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:13.933711 (+ 76us) server_negotiation.cc:407] Connection header received
0811 22:21:13.934437 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:13.934474 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:13.934898 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:13.935218 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:13.935643 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.936610 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.937496 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:13.937967 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:13.938113 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:13.939000 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:13.939006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:13.939018 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:13.939425 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:13.939482 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:13.939503 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:13.939592 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:13.939733 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 22:21:13.939979 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":144,"threads_started":1}
I20260811 22:21:13.941208 14000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4: Neither blocks nor log segments found. Creating new log.
I20260811 22:21:13.942032 14000 log.cc:824] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4: Log is configured to *not* fsync() on all Append() calls
I20260811 22:21:13.943987 14000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4: No bootstrap required, opened a new log
I20260811 22:21:13.947655 14000 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68df8d3bf5ef44f29e8770d738f940d4" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36329 } }
I20260811 22:21:13.947868 14000 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:21:13.947911 14000 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68df8d3bf5ef44f29e8770d738f940d4, State: Initialized, Role: FOLLOWER
I20260811 22:21:13.948294 14000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68df8d3bf5ef44f29e8770d738f940d4" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36329 } }
I20260811 22:21:13.948417 14000 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:21:13.948458 14000 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:21:13.948532 14000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:21:13.949345 14000 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68df8d3bf5ef44f29e8770d738f940d4" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36329 } }
I20260811 22:21:13.949745 14000 leader_election.cc:302] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68df8d3bf5ef44f29e8770d738f940d4; no voters:
I20260811 22:21:13.950033 14000 leader_election.cc:288] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:21:13.950148 14008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:21:13.950400 14008 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [term 1 LEADER]: Becoming Leader. State: Replica: 68df8d3bf5ef44f29e8770d738f940d4, State: Running, Role: LEADER
I20260811 22:21:13.950768 14008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68df8d3bf5ef44f29e8770d738f940d4" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36329 } }
I20260811 22:21:13.951268 14000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:21:13.952370 14010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68df8d3bf5ef44f29e8770d738f940d4. Latest consensus state: current_term: 1 leader_uuid: "68df8d3bf5ef44f29e8770d738f940d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68df8d3bf5ef44f29e8770d738f940d4" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36329 } } }
I20260811 22:21:13.952385 14009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68df8d3bf5ef44f29e8770d738f940d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68df8d3bf5ef44f29e8770d738f940d4" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36329 } } }
I20260811 22:21:13.952515 14009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [sys.catalog]: This master's current role is: LEADER
I20260811 22:21:13.952515 14010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4 [sys.catalog]: This master's current role is: LEADER
I20260811 22:21:13.952962 14020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:21:13.956183 14020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:21:13.962499 14020 catalog_manager.cc:1347] Generated new cluster ID: 29355be927db4360b002b4983a647fdc
I20260811 22:21:13.962569 14020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:21:13.971608 14020 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:21:13.972591 14020 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:21:13.977932 14020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68df8d3bf5ef44f29e8770d738f940d4: Generated new TSK 0
I20260811 22:21:13.978745 14020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:21:13.989627 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36329
--builtin_ntp_servers=127.29.3.84:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 22:21:14.112826 14047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:14.113086 14047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:14.113128 14047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:14.117122 14047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:14.117173 14047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:14.117235 14047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:21:14.121943 14047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:14.123060 14047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:14.124461 14047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:14.133980 14056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.134074 14047 server_base.cc:1034] running on GCE node
W20260811 22:21:14.134130 14064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:14.133983 14058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.134752 14047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:14.135432 14047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:14.136591 14047 hybrid_clock.cc:648] HybridClock initialized: now 1786486874136568 us; error 38 us; skew 500 ppm
I20260811 22:21:14.138464 14047 webserver.cc:460] Webserver started at http://127.29.3.65:32797/ using document root <none> and password file <none>
I20260811 22:21:14.139020 14047 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:14.139087 14047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:14.139304 14047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:14.141297 14047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "9396ad152a784556b1d8173e9aee6026"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "4f711d1f9a69f4ce6a65ea71774257f6"
server_key_iv: "92368d844e336c5fad88a3f8952de78d"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.141824 14047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "9396ad152a784556b1d8173e9aee6026"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "4f711d1f9a69f4ce6a65ea71774257f6"
server_key_iv: "92368d844e336c5fad88a3f8952de78d"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.145418 14047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:21:14.147982 14072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.149143 14047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:21:14.149261 14047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "9396ad152a784556b1d8173e9aee6026"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "4f711d1f9a69f4ce6a65ea71774257f6"
server_key_iv: "92368d844e336c5fad88a3f8952de78d"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.149361 14047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:14.187244 14047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:14.187973 14047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:14.188159 14047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:14.188777 14047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:14.189762 14047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:14.189812 14047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.189867 14047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:14.189895 14047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.198302 14047 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:35909
I20260811 22:21:14.198348 14293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:35909 every 8 connection(s)
I20260811 22:21:14.199208 14047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:21:14.208002 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 14047
I20260811 22:21:14.208109 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:21:14.208184 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.200967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:54749 (local address 127.29.3.126:36329)
0811 22:21:14.201092 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:14.201096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:14.202202 (+ 1106us) server_negotiation.cc:407] Connection header received
0811 22:21:14.203624 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:14.203628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:14.203694 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:14.203773 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:14.204682 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.205231 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.206359 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.206598 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.206692 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:14.207687 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:14.207691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:14.207693 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:14.207707 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:14.207727 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:14.207735 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:14.207795 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:14.207945 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 22:21:14.208021 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:21:14.208421 29709 external_mini_cluster.cc:1428] Setting key 655b3735b043dee4404fc05b5d687ddc
I20260811 22:21:14.209054 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.201352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36329 (local address 127.29.3.65:54749)
0811 22:21:14.202043 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:14.202102 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:14.203406 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:14.203952 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:14.203959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:14.204043 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:14.204531 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:14.204543 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.205353 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.205356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:14.206225 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:14.206235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.206747 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.206751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:14.206907 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:14.207428 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:21:14.207430 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:14.207471 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:21:14.207533 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:14.207806 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:14.207815 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:14.207822 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:21:14.207933 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 22:21:14.208218 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260811 22:21:14.210278 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36329
--builtin_ntp_servers=127.29.3.84:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 22:21:14.210425 14294 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36329
I20260811 22:21:14.210664 14294 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:14.211277 14294 heartbeater.cc:511] Master 127.29.3.126:36329 requested a full tablet report, sending...
I20260811 22:21:14.213013 13935 ts_manager.cc:194] Registered new tserver with Master: 9396ad152a784556b1d8173e9aee6026 (127.29.3.65:35909)
I20260811 22:21:14.214406 13935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.65:54749
W20260811 22:21:14.322484 14302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:14.322736 14302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:14.322770 14302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:14.326656 14302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:14.326700 14302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:14.326749 14302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:21:14.331353 14302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:14.332352 14302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:14.333741 14302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:14.343009 14313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:14.343039 14310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.343222 14302 server_base.cc:1034] running on GCE node
W20260811 22:21:14.343021 14315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.343779 14302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:14.344396 14302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:14.345592 14302 hybrid_clock.cc:648] HybridClock initialized: now 1786486874345567 us; error 40 us; skew 500 ppm
I20260811 22:21:14.347393 14302 webserver.cc:460] Webserver started at http://127.29.3.66:42701/ using document root <none> and password file <none>
I20260811 22:21:14.347952 14302 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:14.348012 14302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:14.348186 14302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:14.350124 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "3c3ccad6f2bf4a8790042350bb99286b"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "e159bb8bb96caec65adee8324ef28a8b"
server_key_iv: "370771285ecebf6f6e4ad56958908e69"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.350684 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "3c3ccad6f2bf4a8790042350bb99286b"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "e159bb8bb96caec65adee8324ef28a8b"
server_key_iv: "370771285ecebf6f6e4ad56958908e69"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.354218 14302 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:21:14.356602 14329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.357795 14302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:21:14.357908 14302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "3c3ccad6f2bf4a8790042350bb99286b"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "e159bb8bb96caec65adee8324ef28a8b"
server_key_iv: "370771285ecebf6f6e4ad56958908e69"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.357998 14302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:14.379354 14302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:14.380005 14302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:14.380169 14302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:14.380736 14302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:14.381661 14302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:14.381708 14302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.381760 14302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:14.381788 14302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.390214 14302 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:37025
I20260811 22:21:14.390254 14538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:37025 every 8 connection(s)
I20260811 22:21:14.391112 14302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:21:14.398427 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 14302
I20260811 22:21:14.398533 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:21:14.398814 29709 external_mini_cluster.cc:1428] Setting key cb7391a1934684ec70f4c21864d8a0a1
I20260811 22:21:14.400525 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.392936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:41259 (local address 127.29.3.126:36329)
0811 22:21:14.393112 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:14.393116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:14.394149 (+ 1033us) server_negotiation.cc:407] Connection header received
0811 22:21:14.395573 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:14.395576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:14.395635 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:14.395712 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:14.396627 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.397157 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.398401 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.398705 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.398795 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:14.399917 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:14.399921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:14.399924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:14.399937 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:14.399958 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:14.399966 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:14.400045 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:14.400176 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 22:21:14.400318 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:21:14.400899 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36329
--builtin_ntp_servers=127.29.3.84:40481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 22:21:14.401402 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.393321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36329 (local address 127.29.3.66:41259)
0811 22:21:14.394002 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:14.394045 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:14.395337 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:14.395905 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:14.395911 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:14.395996 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:14.396479 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:14.396490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.397285 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.397288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:14.398229 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:14.398241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.398864 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.398869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:14.399075 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:14.399658 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:21:14.399660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:14.399700 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:21:14.399766 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:14.400037 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:14.400046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:14.400053 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:21:14.400170 (+ 117us) client_negotiation.cc:240] Negotiation successful
0811 22:21:14.400476 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":124,"threads_started":1}
I20260811 22:21:14.402691 14541 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36329
I20260811 22:21:14.402952 14541 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:14.403561 14541 heartbeater.cc:511] Master 127.29.3.126:36329 requested a full tablet report, sending...
I20260811 22:21:14.404690 13935 ts_manager.cc:194] Registered new tserver with Master: 3c3ccad6f2bf4a8790042350bb99286b (127.29.3.66:37025)
I20260811 22:21:14.405397 13935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.66:41259
W20260811 22:21:14.512732 14547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:14.512980 14547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:14.513015 14547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:14.516965 14547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:14.517020 14547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:14.517068 14547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:21:14.521648 14547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:14.522668 14547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:14.524003 14547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:14.533674 14562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:14.533730 14554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:14.533949 14558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.533806 14547 server_base.cc:1034] running on GCE node
I20260811 22:21:14.534396 14547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:14.535008 14547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:14.536170 14547 hybrid_clock.cc:648] HybridClock initialized: now 1786486874536140 us; error 36 us; skew 500 ppm
I20260811 22:21:14.537931 14547 webserver.cc:460] Webserver started at http://127.29.3.67:42901/ using document root <none> and password file <none>
I20260811 22:21:14.538542 14547 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:14.538609 14547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:14.538820 14547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:14.540895 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "ce27f038c87c4a549c32da0159605afb"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "5f3d8c50cb1f1afe1fa60a7f78d6986c"
server_key_iv: "8ae0505634bc0d8c6c7e02cb587d5ca8"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.541448 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "ce27f038c87c4a549c32da0159605afb"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "5f3d8c50cb1f1afe1fa60a7f78d6986c"
server_key_iv: "8ae0505634bc0d8c6c7e02cb587d5ca8"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.545075 14547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:21:14.547571 14572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.548767 14547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:21:14.548868 14547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "ce27f038c87c4a549c32da0159605afb"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "5f3d8c50cb1f1afe1fa60a7f78d6986c"
server_key_iv: "8ae0505634bc0d8c6c7e02cb587d5ca8"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.548964 14547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:14.565608 14547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:14.566246 14547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:14.566433 14547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:14.567000 14547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:14.567956 14547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:14.568003 14547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.568042 14547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:14.568070 14547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.576297 14547 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:46051
I20260811 22:21:14.576326 14802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:46051 every 8 connection(s)
I20260811 22:21:14.577200 14547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:21:14.578439 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 14547
I20260811 22:21:14.578545 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:21:14.578820 29709 external_mini_cluster.cc:1428] Setting key 7517a67ae13530d4358c205552fcb246
I20260811 22:21:14.586604 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.579169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:41825 (local address 127.29.3.126:36329)
0811 22:21:14.579333 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:14.579337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:14.580480 (+ 1143us) server_negotiation.cc:407] Connection header received
0811 22:21:14.581929 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:14.581932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:14.581987 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:14.582061 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:14.583093 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.583643 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.584774 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.585004 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.585092 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:14.586108 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:14.586112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:14.586114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:14.586129 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:14.586149 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:14.586157 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:14.586224 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:14.586391 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 22:21:14.586458 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:21:14.587455 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.579602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36329 (local address 127.29.3.67:41825)
0811 22:21:14.580352 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:14.580394 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:14.581731 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:14.582236 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:14.582243 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:14.582376 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:14.582957 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:14.582971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.583775 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.583779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:14.584657 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:14.584667 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:14.585146 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:14.585151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:14.585328 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:14.585865 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:21:14.585867 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:14.585906 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:21:14.585969 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:14.586225 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:14.586234 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:14.586241 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:21:14.586388 (+ 147us) client_negotiation.cc:240] Negotiation successful
0811 22:21:14.586674 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260811 22:21:14.588503 14803 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36329
I20260811 22:21:14.588726 14803 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:14.589213 14803 heartbeater.cc:511] Master 127.29.3.126:36329 requested a full tablet report, sending...
I20260811 22:21:14.590205 13935 ts_manager.cc:194] Registered new tserver with Master: ce27f038c87c4a549c32da0159605afb (127.29.3.67:46051)
I20260811 22:21:14.590795 13935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.67:41825
I20260811 22:21:14.592559 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 22:21:14.595216 14003 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:14.594416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.29.3.126:36329)
0811 22:21:14.594527 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:14.594530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:14.594706 (+ 176us) server_negotiation.cc:407] Connection header received
0811 22:21:14.594802 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:14.594804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:14.594842 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:14.594908 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:14.595094 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37560: BlockingRecv error: recv got EOF from 127.0.0.1:37560 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:21:14.595873 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 14047
I20260811 22:21:14.614195 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 14302
I20260811 22:21:14.626967 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 14547
I20260811 22:21:14.648154 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 13876
2026-08-11T22:21:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (937 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T22:21:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:21:14Z Disabled control of system clock
I20260811 22:21:14.680858 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44127
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:44127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 22:21:14.792796 14823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:14.793061 14823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:14.793097 14823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:14.796980 14823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:21:14.797037 14823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:14.797051 14823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:14.797067 14823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:21:14.797080 14823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:21:14.801723 14823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:44127
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:14.802791 14823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:14.804132 14823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:14.812410 14838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:14.812404 14837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.812494 14823 server_base.cc:1034] running on GCE node
W20260811 22:21:14.812500 14844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:14.813194 14823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:14.814244 14823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:14.815415 14823 hybrid_clock.cc:648] HybridClock initialized: now 1786486874815393 us; error 35 us; skew 500 ppm
I20260811 22:21:14.817164 14823 webserver.cc:460] Webserver started at http://127.29.3.126:38823/ using document root <none> and password file <none>
I20260811 22:21:14.817688 14823 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:14.817745 14823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:14.817915 14823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:14.819918 14823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "93c4203459ad42e2b730dbcd28579b44"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "0dfd16616ae300a8cbd05477ce798d68"
server_key_iv: "15b3ebf367d7cde5e8363047a64613b4"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.820410 14823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "93c4203459ad42e2b730dbcd28579b44"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "0dfd16616ae300a8cbd05477ce798d68"
server_key_iv: "15b3ebf367d7cde5e8363047a64613b4"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.823940 14823 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260811 22:21:14.826272 14854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:14.827527 14823 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:21:14.827662 14823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "93c4203459ad42e2b730dbcd28579b44"
format_stamp: "Formatted at 2026-08-11 22:21:14 on dist-test-slave-4tdj"
server_key: "0dfd16616ae300a8cbd05477ce798d68"
server_key_iv: "15b3ebf367d7cde5e8363047a64613b4"
server_key_version: "encryptionkey@0"
I20260811 22:21:14.827755 14823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:14.832175 14823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:14.832757 14823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:14.832892 14823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:14.839586 14823 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:44127
I20260811 22:21:14.839601 14962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:44127 every 8 connection(s)
I20260811 22:21:14.840478 14823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:21:14.843865 14963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:21:14.846532 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 14823
I20260811 22:21:14.846650 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:21:14.846962 29709 external_mini_cluster.cc:1428] Setting key 27d73c4b40c92a82e1fa7e5de453a742
I20260811 22:21:14.849877 14963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44: Bootstrap starting.
I20260811 22:21:14.852227 14966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:14.848338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.29.3.126:44127)
0811 22:21:14.848929 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:14.848944 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:14.848979 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:21:14.849609 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:14.849626 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:14.849969 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:14.850263 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:14.851039 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:14.851043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:14.851054 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:14.851063 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:14.851116 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:14.851131 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:14.851234 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:14.851322 (+ 88us) server_negotiation.cc:299] Negotiation successful
0811 22:21:14.851560 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":126,"threads_started":1}
I20260811 22:21:14.852371 14963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44: Neither blocks nor log segments found. Creating new log.
I20260811 22:21:14.853148 14963 log.cc:824] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44: Log is configured to *not* fsync() on all Append() calls
I20260811 22:21:14.855195 14963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44: No bootstrap required, opened a new log
I20260811 22:21:14.858872 14963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c4203459ad42e2b730dbcd28579b44" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44127 } }
I20260811 22:21:14.859109 14963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:21:14.859143 14963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c4203459ad42e2b730dbcd28579b44, State: Initialized, Role: FOLLOWER
I20260811 22:21:14.859551 14963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c4203459ad42e2b730dbcd28579b44" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44127 } }
I20260811 22:21:14.859684 14963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:21:14.859747 14963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:21:14.859845 14963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:21:14.860836 14963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c4203459ad42e2b730dbcd28579b44" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44127 } }
I20260811 22:21:14.861157 14963 leader_election.cc:302] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93c4203459ad42e2b730dbcd28579b44; no voters:
I20260811 22:21:14.861464 14963 leader_election.cc:288] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:21:14.861584 14968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:21:14.861819 14968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [term 1 LEADER]: Becoming Leader. State: Replica: 93c4203459ad42e2b730dbcd28579b44, State: Running, Role: LEADER
I20260811 22:21:14.862154 14968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c4203459ad42e2b730dbcd28579b44" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44127 } }
I20260811 22:21:14.862710 14963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:21:14.863787 14970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93c4203459ad42e2b730dbcd28579b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c4203459ad42e2b730dbcd28579b44" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44127 } } }
I20260811 22:21:14.863776 14972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93c4203459ad42e2b730dbcd28579b44. Latest consensus state: current_term: 1 leader_uuid: "93c4203459ad42e2b730dbcd28579b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93c4203459ad42e2b730dbcd28579b44" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44127 } } }
I20260811 22:21:14.863924 14970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [sys.catalog]: This master's current role is: LEADER
I20260811 22:21:14.863933 14972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44 [sys.catalog]: This master's current role is: LEADER
I20260811 22:21:14.864315 14988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:21:14.867077 14988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:21:14.872889 14988 catalog_manager.cc:1347] Generated new cluster ID: 5a07b504bcf14e99b8ec970cec7222b0
I20260811 22:21:14.872963 14988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:21:14.895578 14988 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:21:14.895948 14988 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:21:14.920630 14988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93c4203459ad42e2b730dbcd28579b44: Generated new TSK 0
I20260811 22:21:14.921409 14988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:21:14.934056 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44127
--builtin_ntp_servers=127.29.3.84:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 22:21:15.049571 15010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:15.049836 15010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:15.049891 15010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:15.053840 15010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:15.053898 15010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:15.053973 15010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:21:15.058706 15010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:15.059798 15010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:15.061151 15010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:15.071043 15016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:15.070988 15017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:15.070986 15025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:15.071357 15010 server_base.cc:1034] running on GCE node
I20260811 22:21:15.071864 15010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:15.072526 15010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:15.073698 15010 hybrid_clock.cc:648] HybridClock initialized: now 1786486875073666 us; error 45 us; skew 500 ppm
I20260811 22:21:15.075529 15010 webserver.cc:460] Webserver started at http://127.29.3.65:44847/ using document root <none> and password file <none>
I20260811 22:21:15.076114 15010 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:15.076184 15010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:15.076400 15010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:15.078528 15010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "a84bb61a7b6f47479a2cd050ab640a69"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "09db1afc5de1fcb536028fb88d786280"
server_key_iv: "258aa834bfed8000d4acd037543ee947"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.079084 15010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "a84bb61a7b6f47479a2cd050ab640a69"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "09db1afc5de1fcb536028fb88d786280"
server_key_iv: "258aa834bfed8000d4acd037543ee947"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.082715 15010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:21:15.085314 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.086544 15010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:21:15.086647 15010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "a84bb61a7b6f47479a2cd050ab640a69"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "09db1afc5de1fcb536028fb88d786280"
server_key_iv: "258aa834bfed8000d4acd037543ee947"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.086764 15010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:15.091269 15010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:15.091866 15010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:15.092017 15010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:15.092537 15010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:15.093431 15010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:15.093478 15010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.093519 15010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:15.093547 15010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.101917 15010 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:43599
I20260811 22:21:15.101944 15263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:43599 every 8 connection(s)
I20260811 22:21:15.102855 15010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:21:15.108920 14966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:15.104647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:45137 (local address 127.29.3.126:44127)
0811 22:21:15.104795 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:15.104800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:15.105928 (+ 1128us) server_negotiation.cc:407] Connection header received
0811 22:21:15.107459 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:15.107462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:15.107502 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:15.107574 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:15.108550 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:15.108553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:15.108555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:15.108566 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:15.108586 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:15.108594 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:15.108668 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:15.108767 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 22:21:15.108806 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:21:15.109757 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:15.105102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44127 (local address 127.29.3.65:45137)
0811 22:21:15.105783 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:15.105825 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:15.107233 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:15.107726 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:15.107733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:15.107823 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:15.108307 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:21:15.108309 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:15.108350 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:21:15.108418 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:15.108656 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:15.108660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:15.108664 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:21:15.108797 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 22:21:15.108992 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":126,"threads_started":1}
I20260811 22:21:15.110934 15265 heartbeater.cc:347] Connected to a master server at 127.29.3.126:44127
I20260811 22:21:15.111157 15265 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:15.111595 15265 heartbeater.cc:511] Master 127.29.3.126:44127 requested a full tablet report, sending...
I20260811 22:21:15.111719 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 15010
I20260811 22:21:15.111828 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:21:15.112123 29709 external_mini_cluster.cc:1428] Setting key 23f130d677cbd69f1c28a592a75248aa
I20260811 22:21:15.113310 14885 ts_manager.cc:194] Registered new tserver with Master: a84bb61a7b6f47479a2cd050ab640a69 (127.29.3.65:43599)
I20260811 22:21:15.114200 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44127
--builtin_ntp_servers=127.29.3.84:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 22:21:15.225680 15270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:15.225921 15270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:15.225958 15270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:15.229848 15270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:15.229899 15270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:15.229950 15270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:21:15.234637 15270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:15.235617 15270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:15.236967 15270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:15.246424 15277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:15.246407 15283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:15.246407 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:15.246461 15270 server_base.cc:1034] running on GCE node
I20260811 22:21:15.247109 15270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:15.247722 15270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:15.248875 15270 hybrid_clock.cc:648] HybridClock initialized: now 1786486875248858 us; error 29 us; skew 500 ppm
I20260811 22:21:15.250651 15270 webserver.cc:460] Webserver started at http://127.29.3.66:37835/ using document root <none> and password file <none>
I20260811 22:21:15.251220 15270 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:15.251284 15270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:15.251492 15270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:15.253517 15270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "de86e384070444969ad533e88d872bf5"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "9b82dcf8b76205c6bda92edb4dae5d16"
server_key_iv: "3b2733ac769339c59c8c8d3e65cf3d1b"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.254055 15270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "de86e384070444969ad533e88d872bf5"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "9b82dcf8b76205c6bda92edb4dae5d16"
server_key_iv: "3b2733ac769339c59c8c8d3e65cf3d1b"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.257648 15270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:21:15.260133 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.261272 15270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:21:15.261384 15270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "de86e384070444969ad533e88d872bf5"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "9b82dcf8b76205c6bda92edb4dae5d16"
server_key_iv: "3b2733ac769339c59c8c8d3e65cf3d1b"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.261483 15270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:15.266101 15270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:15.266727 15270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:15.266886 15270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:15.267405 15270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:15.268290 15270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:15.268338 15270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.268389 15270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:15.268415 15270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.276501 15270 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:46117
I20260811 22:21:15.276533 15504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:46117 every 8 connection(s)
I20260811 22:21:15.277348 15270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:21:15.280652 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 15270
I20260811 22:21:15.280781 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:21:15.281080 29709 external_mini_cluster.cc:1428] Setting key b1a8f6d29d482fec978304f16784773c
I20260811 22:21:15.283186 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44127
--builtin_ntp_servers=127.29.3.84:40727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 22:21:15.283681 14966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:15.279161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:50039 (local address 127.29.3.126:44127)
0811 22:21:15.279315 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:15.279320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:15.280330 (+ 1010us) server_negotiation.cc:407] Connection header received
0811 22:21:15.281913 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:15.281916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:15.281952 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:15.282035 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:15.283165 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:15.283168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:15.283170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:15.283182 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:15.283202 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:15.283209 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:15.283295 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:15.283446 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 22:21:15.283484 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:21:15.284644 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:15.279529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44127 (local address 127.29.3.66:50039)
0811 22:21:15.280183 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:15.280225 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:15.281695 (+ 1470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:15.282177 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:15.282184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:15.282270 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:15.282848 (+ 578us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:21:15.282850 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:15.282903 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:21:15.282992 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:15.283295 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:15.283304 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:15.283311 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:21:15.283478 (+ 167us) client_negotiation.cc:240] Negotiation successful
0811 22:21:15.283714 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260811 22:21:15.285792 15505 heartbeater.cc:347] Connected to a master server at 127.29.3.126:44127
I20260811 22:21:15.286003 15505 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:15.286428 15505 heartbeater.cc:511] Master 127.29.3.126:44127 requested a full tablet report, sending...
I20260811 22:21:15.287434 14885 ts_manager.cc:194] Registered new tserver with Master: de86e384070444969ad533e88d872bf5 (127.29.3.66:46117)
W20260811 22:21:15.397027 15511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:15.397303 15511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:15.397341 15511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:15.401288 15511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:15.401340 15511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:15.401393 15511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:21:15.406080 15511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:15.407169 15511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:15.408500 15511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:15.417688 15522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:15.417758 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:15.417845 15523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:15.417909 15511 server_base.cc:1034] running on GCE node
I20260811 22:21:15.418466 15511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:15.419085 15511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:15.420248 15511 hybrid_clock.cc:648] HybridClock initialized: now 1786486875420230 us; error 28 us; skew 500 ppm
I20260811 22:21:15.421975 15511 webserver.cc:460] Webserver started at http://127.29.3.67:44207/ using document root <none> and password file <none>
I20260811 22:21:15.422617 15511 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:15.422683 15511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:15.422850 15511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:15.424853 15511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "4b393d30ebbb445c942cb9d4bb07b6b6"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "9b6f0528dffe6a3a76369d44f382ff0f"
server_key_iv: "2456f95330505216f85cde66801e0c50"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.425385 15511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "4b393d30ebbb445c942cb9d4bb07b6b6"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "9b6f0528dffe6a3a76369d44f382ff0f"
server_key_iv: "2456f95330505216f85cde66801e0c50"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.429044 15511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 22:21:15.431639 15546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.432871 15511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:21:15.432986 15511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "4b393d30ebbb445c942cb9d4bb07b6b6"
format_stamp: "Formatted at 2026-08-11 22:21:15 on dist-test-slave-4tdj"
server_key: "9b6f0528dffe6a3a76369d44f382ff0f"
server_key_iv: "2456f95330505216f85cde66801e0c50"
server_key_version: "encryptionkey@0"
I20260811 22:21:15.433090 15511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:15.437505 15511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:15.438058 15511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:15.438189 15511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:15.438696 15511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:15.439564 15511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:15.439610 15511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.439646 15511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:15.439667 15511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:15.447629 15511 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:41173
I20260811 22:21:15.447700 15742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:41173 every 8 connection(s)
I20260811 22:21:15.448511 15511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:21:15.450445 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 15511
I20260811 22:21:15.450546 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:21:15.450815 29709 external_mini_cluster.cc:1428] Setting key b1452f02f5d440105c1cb76ed9a8d525
I20260811 22:21:15.454739 14966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:15.450345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43745 (local address 127.29.3.126:44127)
0811 22:21:15.450493 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:15.450499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:15.451775 (+ 1276us) server_negotiation.cc:407] Connection header received
0811 22:21:15.453251 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:15.453254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:15.453293 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:15.453355 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:15.454367 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:15.454371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:15.454374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:21:15.454388 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:15.454409 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:21:15.454418 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:15.454489 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:15.454592 (+ 103us) server_negotiation.cc:299] Negotiation successful
0811 22:21:15.454628 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:21:15.455606 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:15.450814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44127 (local address 127.29.3.67:43745)
0811 22:21:15.451614 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:15.451670 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:15.453062 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:15.453493 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:15.453500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:15.453592 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:15.454091 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:21:15.454093 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:15.454134 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:21:15.454204 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:15.454484 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:15.454489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:15.454494 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:21:15.454627 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 22:21:15.454845 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":115,"threads_started":1}
I20260811 22:21:15.456636 15745 heartbeater.cc:347] Connected to a master server at 127.29.3.126:44127
I20260811 22:21:15.456873 15745 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:15.457278 15745 heartbeater.cc:511] Master 127.29.3.126:44127 requested a full tablet report, sending...
I20260811 22:21:15.458261 14885 ts_manager.cc:194] Registered new tserver with Master: 4b393d30ebbb445c942cb9d4bb07b6b6 (127.29.3.67:41173)
I20260811 22:21:15.464596 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 22:21:15.467386 14966 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:21:15.466553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.29.3.126:44127)
0811 22:21:15.466673 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:15.466677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:15.466866 (+ 189us) server_negotiation.cc:407] Connection header received
0811 22:21:15.466983 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:15.466985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:15.467021 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:15.467076 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:15.467256 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50290: BlockingRecv error: recv got EOF from 127.0.0.1:50290 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:21:15.467836 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 15010
I20260811 22:21:15.478760 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 15270
I20260811 22:21:15.514629 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 15511
I20260811 22:21:15.527110 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 14823
2026-08-11T22:21:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (881 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:21:15 dist-test-slave-4tdj krb5kdc[15765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:21:15 dist-test-slave-4tdj krb5kdc[15765](info): set up 2 sockets
Aug 11 22:21:15 dist-test-slave-4tdj krb5kdc[15765](info): commencing operation
krb5kdc: starting...
W20260811 22:21:16.713218 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:21:16 dist-test-slave-4tdj krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:21:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:21:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:21:16.837786 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:43269
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:43269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:21:16.949901 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:16.950198 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:16.950258 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:16.954355 15784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:21:16.954419 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:16.954443 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:16.954468 15784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:21:16.954496 15784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:21:16.959889 15784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:43269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:43269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:16.961000 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:16.962045 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:16.970597 15792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:16.970618 15803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:16.970656 15784 server_base.cc:1034] running on GCE node
W20260811 22:21:16.970609 15790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:16.971432 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:16.972522 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:16.973683 15784 hybrid_clock.cc:648] HybridClock initialized: now 1786486876973661 us; error 41 us; skew 500 ppm
Aug 11 22:21:16 dist-test-slave-4tdj krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486876, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:21:16.976294 15784 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:21:16.977129 15784 webserver.cc:460] Webserver started at http://127.29.3.126:37367/ using document root <none> and password file <none>
I20260811 22:21:16.977671 15784 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:16.977737 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:16.977926 15784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:16.979982 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "891899724c78456c9984ef15c3636340"
format_stamp: "Formatted at 2026-08-11 22:21:16 on dist-test-slave-4tdj"
server_key: "1bafecb01a582d73c66b8d2094fe0867"
server_key_iv: "74e1db40e872e392a11a3da058b53707"
server_key_version: "encryptionkey@0"
I20260811 22:21:16.980531 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "891899724c78456c9984ef15c3636340"
format_stamp: "Formatted at 2026-08-11 22:21:16 on dist-test-slave-4tdj"
server_key: "1bafecb01a582d73c66b8d2094fe0867"
server_key_iv: "74e1db40e872e392a11a3da058b53707"
server_key_version: "encryptionkey@0"
I20260811 22:21:16.984234 15784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260811 22:21:16.986610 15814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:16.987888 15784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:21:16.988004 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "891899724c78456c9984ef15c3636340"
format_stamp: "Formatted at 2026-08-11 22:21:16 on dist-test-slave-4tdj"
server_key: "1bafecb01a582d73c66b8d2094fe0867"
server_key_iv: "74e1db40e872e392a11a3da058b53707"
server_key_version: "encryptionkey@0"
I20260811 22:21:16.988098 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:17.012557 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:17.015316 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:17.015509 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:17.023399 15784 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:43269
I20260811 22:21:17.023770 15907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:43269 every 8 connection(s)
I20260811 22:21:17.024672 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:21:17.028497 15908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:21:17.034286 15908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340: Bootstrap starting.
I20260811 22:21:17.034660 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 15784
I20260811 22:21:17.034768 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:21:17.035012 29709 external_mini_cluster.cc:1428] Setting key 3185c69a30720759ec41a70abed4224d
I20260811 22:21:17.036988 15908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340: Neither blocks nor log segments found. Creating new log.
I20260811 22:21:17.037914 15908 log.cc:824] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340: Log is configured to *not* fsync() on all Append() calls
I20260811 22:21:17.040217 15908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340: No bootstrap required, opened a new log
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:21:17.044139 15908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891899724c78456c9984ef15c3636340" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43269 } }
I20260811 22:21:17.044382 15908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:21:17.044443 15908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 891899724c78456c9984ef15c3636340, State: Initialized, Role: FOLLOWER
I20260811 22:21:17.044920 15908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891899724c78456c9984ef15c3636340" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43269 } }
I20260811 22:21:17.045101 15908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:21:17.045177 15908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:21:17.045307 15908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:21:17.046509 15908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891899724c78456c9984ef15c3636340" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43269 } }
I20260811 22:21:17.046890 15908 leader_election.cc:302] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 891899724c78456c9984ef15c3636340; no voters:
I20260811 22:21:17.047245 15908 leader_election.cc:288] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:21:17.047338 15920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:21:17.047526 15920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [term 1 LEADER]: Becoming Leader. State: Replica: 891899724c78456c9984ef15c3636340, State: Running, Role: LEADER
I20260811 22:21:17.047842 15920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891899724c78456c9984ef15c3636340" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43269 } }
I20260811 22:21:17.048599 15908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:21:17.049840 15922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "891899724c78456c9984ef15c3636340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891899724c78456c9984ef15c3636340" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43269 } } }
I20260811 22:21:17.049997 15922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [sys.catalog]: This master's current role is: LEADER
I20260811 22:21:17.049855 15925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 891899724c78456c9984ef15c3636340. Latest consensus state: current_term: 1 leader_uuid: "891899724c78456c9984ef15c3636340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "891899724c78456c9984ef15c3636340" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43269 } } }
I20260811 22:21:17.050141 15925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340 [sys.catalog]: This master's current role is: LEADER
I20260811 22:21:17.050271 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.036343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.29.3.126:43269)
0811 22:21:17.036770 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.036791 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.036836 (+ 45us) server_negotiation.cc:407] Connection header received
0811 22:21:17.037719 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.037742 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.038176 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.038547 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:17.039302 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.040750 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.041587 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.042320 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.042496 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.044851 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:17.044858 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:17.044874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:21:17.044907 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:17.046907 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.047395 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.047401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.047408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.047554 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.047901 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.047904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.047905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.048336 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:17.048528 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.048931 (+ 403us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.049270 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
I20260811 22:21:17.050506 15936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:21:17.053650 15936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:21:17.059223 15936 catalog_manager.cc:1347] Generated new cluster ID: 3c89e04657744c0e9af7de0abb485df7
I20260811 22:21:17.059299 15936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:21:17.065335 15936 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:21:17.066637 15936 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:21:17.071394 15936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340: Generated new TSK 0
I20260811 22:21:17.072154 15936 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:21:17.104704 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43269
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:21:17.217374 15960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:17.217634 15960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:17.217675 15960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:17.221540 15960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:17.221589 15960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:17.221643 15960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:21:17.226747 15960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:17.227690 15960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:17.228632 15960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:17.238067 15975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:17.238097 15973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:17.238086 15971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:17.238348 15960 server_base.cc:1034] running on GCE node
I20260811 22:21:17.238790 15960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:17.239389 15960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:17.240548 15960 hybrid_clock.cc:648] HybridClock initialized: now 1786486877240524 us; error 38 us; skew 500 ppm
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:21:17.243208 15960 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:21:17.244024 15960 webserver.cc:460] Webserver started at http://127.29.3.65:38317/ using document root <none> and password file <none>
I20260811 22:21:17.244545 15960 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:17.244607 15960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:17.244819 15960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:17.246856 15960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "1b905dacb3184bbf88bbb9d80609be06"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "3c5c2e80a214a6696d8bbd126a5201d2"
server_key_iv: "f25278efb0dc89c050ef77eb01e6679e"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.247387 15960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "1b905dacb3184bbf88bbb9d80609be06"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "3c5c2e80a214a6696d8bbd126a5201d2"
server_key_iv: "f25278efb0dc89c050ef77eb01e6679e"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.251188 15960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:21:17.253522 15984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.254737 15960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:21:17.254845 15960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "1b905dacb3184bbf88bbb9d80609be06"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "3c5c2e80a214a6696d8bbd126a5201d2"
server_key_iv: "f25278efb0dc89c050ef77eb01e6679e"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.254943 15960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:17.265897 15960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:17.268991 15960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:17.269164 15960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:17.269757 15960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:17.270615 15960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:17.270673 15960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.270723 15960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:17.270751 15960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.278978 15960 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:34803
I20260811 22:21:17.278999 16183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:34803 every 8 connection(s)
I20260811 22:21:17.279865 15960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:21:17.281445 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 15960
I20260811 22:21:17.281556 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:21:17.281870 29709 external_mini_cluster.cc:1428] Setting key 167604aa883e8c4347a1973840782bf8
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:21:17.294411 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.281680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:49193 (local address 127.29.3.126:43269)
0811 22:21:17.281837 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.281842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.282742 (+ 900us) server_negotiation.cc:407] Connection header received
0811 22:21:17.284430 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.284433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.284478 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.284535 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:17.286013 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.286823 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.288049 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.288355 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.288462 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.290898 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:17.290903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:17.290906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:21:17.290937 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:17.292422 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.292930 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.292935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.292936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.293031 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.293347 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.293351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.293352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.293571 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:17.293674 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.294146 (+ 472us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.294252 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:21:17.295238 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.282079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:43269 (local address 127.29.3.65:49193)
0811 22:21:17.282580 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.282637 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.284179 (+ 1542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.284721 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.284731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.285237 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:17.285866 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.285883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.286931 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.286933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.287800 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.287810 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.288519 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.288524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.288720 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.289277 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:17.289294 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:17.290710 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:17.292535 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:21:17.292540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:21:17.292552 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:21:17.292828 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:21:17.293119 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:21:17.293121 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:21:17.293122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:21:17.293258 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:21:17.293665 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:17.293671 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:17.293920 (+ 249us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.294078 (+ 158us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.294380 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":153,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:21:17.296510 16185 heartbeater.cc:347] Connected to a master server at 127.29.3.126:43269
I20260811 22:21:17.296748 16185 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:17.297252 16185 heartbeater.cc:511] Master 127.29.3.126:43269 requested a full tablet report, sending...
I20260811 22:21:17.298972 15845 ts_manager.cc:194] Registered new tserver with Master: 1b905dacb3184bbf88bbb9d80609be06 (127.29.3.65:34803)
I20260811 22:21:17.300233 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:49193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:21:17.311236 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43269
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:21:17.423964 16191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:17.424250 16191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:17.424299 16191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:17.428368 16191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:17.428431 16191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:17.428537 16191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:21:17.433823 16191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:17.434937 16191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:17.435905 16191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:17.445292 16199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:17.445324 16202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:17.445279 16205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:17.445358 16191 server_base.cc:1034] running on GCE node
I20260811 22:21:17.445896 16191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:17.446547 16191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:17.447705 16191 hybrid_clock.cc:648] HybridClock initialized: now 1786486877447688 us; error 28 us; skew 500 ppm
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:21:17.450376 16191 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:21:17.451237 16191 webserver.cc:460] Webserver started at http://127.29.3.66:34563/ using document root <none> and password file <none>
I20260811 22:21:17.451797 16191 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:17.451864 16191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:17.452054 16191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:17.454010 16191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "927f0df6047647f4a7c8f8867f37cf7b"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "ec8c6d068c618999f4d8d673d7b4592f"
server_key_iv: "4e9a580872a7015ef24ad9c908a9d757"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.454561 16191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "927f0df6047647f4a7c8f8867f37cf7b"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "ec8c6d068c618999f4d8d673d7b4592f"
server_key_iv: "4e9a580872a7015ef24ad9c908a9d757"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.458088 16191 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:21:17.460613 16215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.461809 16191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:21:17.461928 16191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "927f0df6047647f4a7c8f8867f37cf7b"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "ec8c6d068c618999f4d8d673d7b4592f"
server_key_iv: "4e9a580872a7015ef24ad9c908a9d757"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.462018 16191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:17.484553 16191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:17.487390 16191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:17.487565 16191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:17.488140 16191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:17.489001 16191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:17.489051 16191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.489105 16191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:17.489132 16191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.497566 16191 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:39683
I20260811 22:21:17.497594 16408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:39683 every 8 connection(s)
I20260811 22:21:17.498508 16191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:21:17.508370 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 16191
I20260811 22:21:17.508479 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:21:17.508808 29709 external_mini_cluster.cc:1428] Setting key c6a6472ca64ba3b3def2fc59fd9e7305
I20260811 22:21:17.511737 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.500278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:56907 (local address 127.29.3.126:43269)
0811 22:21:17.500411 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.500414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.501217 (+ 803us) server_negotiation.cc:407] Connection header received
0811 22:21:17.502611 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.502615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.502675 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.502763 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:17.503927 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.504462 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.505595 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.505831 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.505922 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.508205 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:17.508208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:17.508211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:21:17.508238 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:17.509595 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.510124 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.510127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.510128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.510221 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.510644 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.510647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.510648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.510850 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:17.510932 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.511508 (+ 576us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.511609 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:21:17.512770 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.500628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:43269 (local address 127.29.3.66:56907)
0811 22:21:17.501073 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.501116 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.502406 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.502941 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.502948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.503314 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:17.503784 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.503798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.504583 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.504586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.505476 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.505487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.505972 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.505976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.506137 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.506687 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:17.506703 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:17.508042 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:17.509716 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:21:17.509722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:21:17.509734 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:21:17.510009 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:21:17.510363 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:21:17.510365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:21:17.510367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:21:17.510547 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:21:17.510957 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:17.510963 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:17.511276 (+ 313us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.511473 (+ 197us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.511811 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":117,"threads_started":1}
I20260811 22:21:17.513875 16409 heartbeater.cc:347] Connected to a master server at 127.29.3.126:43269
I20260811 22:21:17.514104 16409 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:17.514624 16409 heartbeater.cc:511] Master 127.29.3.126:43269 requested a full tablet report, sending...
I20260811 22:21:17.515686 15845 ts_manager.cc:194] Registered new tserver with Master: 927f0df6047647f4a7c8f8867f37cf7b (127.29.3.66:39683)
I20260811 22:21:17.516450 15845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:56907
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:21:17.535313 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43269
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:38819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:21:17.647539 16415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:21:17.647800 16415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:21:17.647845 16415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:21:17.651943 16415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:21:17.652001 16415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:21:17.652069 16415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:21:17.657312 16415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:38819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:21:17.658421 16415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:21:17.659469 16415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:21:17.669533 16425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:21:17.669528 16426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:17.669821 16415 server_base.cc:1034] running on GCE node
W20260811 22:21:17.669957 16428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:21:17.670421 16415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:21:17.671021 16415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:21:17.673961 16415 hybrid_clock.cc:648] HybridClock initialized: now 1786486877673917 us; error 68 us; skew 500 ppm
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:21:17.677176 16415 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:21:17.678021 16415 webserver.cc:460] Webserver started at http://127.29.3.67:41257/ using document root <none> and password file <none>
I20260811 22:21:17.678606 16415 fs_manager.cc:359] Metadata directory not provided
I20260811 22:21:17.678666 16415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:21:17.678835 16415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:21:17.680810 16415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "490d3eb868704c83afbf32e33ac68beb"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "8c53ad34cd782f85cdeaa009cd88fa6b"
server_key_iv: "b9fb097b53874295f4788e4480282694"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.681366 16415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "490d3eb868704c83afbf32e33ac68beb"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "8c53ad34cd782f85cdeaa009cd88fa6b"
server_key_iv: "b9fb097b53874295f4788e4480282694"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.685091 16415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260811 22:21:17.687608 16443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.688891 16415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:21:17.688997 16415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "490d3eb868704c83afbf32e33ac68beb"
format_stamp: "Formatted at 2026-08-11 22:21:17 on dist-test-slave-4tdj"
server_key: "8c53ad34cd782f85cdeaa009cd88fa6b"
server_key_iv: "b9fb097b53874295f4788e4480282694"
server_key_version: "encryptionkey@0"
I20260811 22:21:17.689087 16415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:21:17.714547 16415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:21:17.717746 16415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:21:17.717909 16415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:21:17.718544 16415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:21:17.719404 16415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:21:17.719455 16415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.719494 16415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:21:17.719515 16415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:21:17.728010 16415 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:40229
I20260811 22:21:17.728034 16638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:40229 every 8 connection(s)
I20260811 22:21:17.728931 16415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:21:17.731868 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 16415
I20260811 22:21:17.731989 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:21:17.732259 29709 external_mini_cluster.cc:1428] Setting key a679871ee75205afe7c08a23e7a2d041
Aug 11 22:21:17 dist-test-slave-4tdj krb5kdc[15765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486877, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:21:17.742386 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.730703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:42199 (local address 127.29.3.126:43269)
0811 22:21:17.730842 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.730846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.731700 (+ 854us) server_negotiation.cc:407] Connection header received
0811 22:21:17.733244 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.733247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.733307 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.733379 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:17.734839 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.735377 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.736511 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.736737 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.736825 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.739229 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:17.739233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:17.739235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:21:17.739261 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:17.740356 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.740835 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.740839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.740840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.740920 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.741240 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.741244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.741245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.741434 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:17.741521 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.742036 (+ 515us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.742167 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:21:17.743191 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.731070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:43269 (local address 127.29.3.67:42199)
0811 22:21:17.731539 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.731583 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.733010 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.733558 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.733565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.734058 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:21:17.734664 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.734679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.735498 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.735501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.736385 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.736395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.736878 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.736882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.737058 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.737570 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:21:17.737587 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:21:17.739047 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:21:17.740465 (+ 1418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:21:17.740471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:21:17.740487 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:21:17.740740 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:21:17.741001 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:21:17.741004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:21:17.741007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:21:17.741145 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:21:17.741535 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:21:17.741542 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:21:17.741804 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.741976 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.742342 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":124,"threads_started":1}
I20260811 22:21:17.744293 16639 heartbeater.cc:347] Connected to a master server at 127.29.3.126:43269
I20260811 22:21:17.744517 16639 heartbeater.cc:464] Registering TS with master...
I20260811 22:21:17.745046 16639 heartbeater.cc:511] Master 127.29.3.126:43269 requested a full tablet report, sending...
I20260811 22:21:17.746104 15844 ts_manager.cc:194] Registered new tserver with Master: 490d3eb868704c83afbf32e33ac68beb (127.29.3.67:40229)
I20260811 22:21:17.746721 15844 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:42199
I20260811 22:21:17.757836 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:21:17.767352 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.759747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.29.3.126:43269)
0811 22:21:17.759861 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.759865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.760023 (+ 158us) server_negotiation.cc:407] Connection header received
0811 22:21:17.760128 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.760130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.760170 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.760242 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:21:17.760873 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.761368 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.762115 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.762358 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.762428 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.763861 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:21:17.763864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:21:17.763867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:21:17.763894 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:21:17.765552 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.765995 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.765998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.766001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.766097 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:21:17.766472 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:21:17.766475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:21:17.766477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:21:17.766718 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:21:17.766839 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.767073 (+ 234us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.767184 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:21:17.770169 15844 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:21:17.773193 15844 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:21:17.787658 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.782953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.29.3.67:40229)
0811 22:21:17.783353 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.783364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.783441 (+ 77us) server_negotiation.cc:407] Connection header received
0811 22:21:17.783562 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.783567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.783749 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.783884 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:21:17.784077 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.785054 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.786402 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.786938 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.787039 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.787189 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.787394 (+ 205us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.787492 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":104,"threads_started":1}
I20260811 22:21:17.787657 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.781452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39683 (local address 127.0.0.1:48868)
0811 22:21:17.781951 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.782011 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.782147 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.782990 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.782995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.783016 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:21:17.783148 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.783155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.784522 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.784526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.785779 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.785789 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.787058 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.787062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.787180 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.787248 (+ 68us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.787355 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.787502 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":93,"threads_started":1}
I20260811 22:21:17.787653 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.782552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:40229 (local address 127.0.0.1:49190)
0811 22:21:17.783367 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.783379 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.783462 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.783870 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.783873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.783892 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:21:17.783986 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.783992 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.785198 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.785202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.786271 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.786279 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.787063 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.787067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.787207 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.787267 (+ 60us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.787351 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.787492 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":57,"thread_start_us":96,"threads_started":1}
I20260811 22:21:17.787747 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.782376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.29.3.65:34803)
0811 22:21:17.782875 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.782889 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.783071 (+ 182us) server_negotiation.cc:407] Connection header received
0811 22:21:17.783196 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.783201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.783421 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.783568 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:21:17.783775 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.784630 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.786147 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.786966 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.787065 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.787203 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.787502 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.787599 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":96,"threads_started":1}
I20260811 22:21:17.788044 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.782187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34803 (local address 127.0.0.1:36402)
0811 22:21:17.782967 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.782980 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.783089 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.783567 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.783570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.783582 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:21:17.783675 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.783681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.784777 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.784780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.786023 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.786034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.787264 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.787268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.787346 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.787368 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.787422 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.787929 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":428,"thread_start_us":116,"threads_started":1}
I20260811 22:21:17.788365 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.781625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.29.3.66:39683)
0811 22:21:17.781949 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.781962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.782317 (+ 355us) server_negotiation.cc:407] Connection header received
0811 22:21:17.782406 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.782411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.782630 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.782793 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:21:17.783257 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.784395 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.785894 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.786684 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.786797 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.786977 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.787862 (+ 885us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.787959 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260811 22:21:17.791038 16298 tablet_service.cc:1467] Processing CreateTablet for tablet 6182432b6a2348e9ac65f94f72895fab (DEFAULT_TABLE table=test-table [id=a561a6f02cd2403887527ddcf0808593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:21:17.791124 16079 tablet_service.cc:1467] Processing CreateTablet for tablet 6182432b6a2348e9ac65f94f72895fab (DEFAULT_TABLE table=test-table [id=a561a6f02cd2403887527ddcf0808593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:21:17.791684 16528 tablet_service.cc:1467] Processing CreateTablet for tablet 6182432b6a2348e9ac65f94f72895fab (DEFAULT_TABLE table=test-table [id=a561a6f02cd2403887527ddcf0808593]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:21:17.792152 16298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6182432b6a2348e9ac65f94f72895fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:21:17.792213 16079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6182432b6a2348e9ac65f94f72895fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:21:17.792605 16528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6182432b6a2348e9ac65f94f72895fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:21:17.798731 16670 tablet_bootstrap.cc:492] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Bootstrap starting.
I20260811 22:21:17.800093 16671 tablet_bootstrap.cc:492] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06: Bootstrap starting.
I20260811 22:21:17.801343 16670 tablet_bootstrap.cc:654] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Neither blocks nor log segments found. Creating new log.
I20260811 22:21:17.801859 16671 tablet_bootstrap.cc:654] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06: Neither blocks nor log segments found. Creating new log.
I20260811 22:21:17.802100 16670 log.cc:824] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:21:17.802647 16673 tablet_bootstrap.cc:492] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Bootstrap starting.
I20260811 22:21:17.802651 16671 log.cc:824] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06: Log is configured to *not* fsync() on all Append() calls
I20260811 22:21:17.804081 16670 tablet_bootstrap.cc:492] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: No bootstrap required, opened a new log
I20260811 22:21:17.804287 16670 ts_tablet_manager.cc:1397] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:21:17.804664 16673 tablet_bootstrap.cc:654] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Neither blocks nor log segments found. Creating new log.
I20260811 22:21:17.805678 16673 log.cc:824] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Log is configured to *not* fsync() on all Append() calls
I20260811 22:21:17.807574 16670 raft_consensus.cc:348] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.807853 16671 tablet_bootstrap.cc:492] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06: No bootstrap required, opened a new log
I20260811 22:21:17.807878 16670 raft_consensus.cc:374] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:21:17.807934 16670 raft_consensus.cc:729] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927f0df6047647f4a7c8f8867f37cf7b, State: Initialized, Role: FOLLOWER
I20260811 22:21:17.808071 16671 ts_tablet_manager.cc:1397] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260811 22:21:17.808456 16670 consensus_queue.cc:260] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.809206 16409 heartbeater.cc:503] Master 127.29.3.126:43269 was elected leader, sending a full tablet report...
I20260811 22:21:17.809234 16673 tablet_bootstrap.cc:492] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: No bootstrap required, opened a new log
I20260811 22:21:17.809407 16673 ts_tablet_manager.cc:1397] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 22:21:17.809530 16670 ts_tablet_manager.cc:1428] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 22:21:17.812150 16671 raft_consensus.cc:348] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.812458 16671 raft_consensus.cc:374] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:21:17.812520 16671 raft_consensus.cc:729] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b905dacb3184bbf88bbb9d80609be06, State: Initialized, Role: FOLLOWER
I20260811 22:21:17.812863 16673 raft_consensus.cc:348] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.813067 16673 raft_consensus.cc:374] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:21:17.812987 16671 consensus_queue.cc:260] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.813118 16673 raft_consensus.cc:729] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 490d3eb868704c83afbf32e33ac68beb, State: Initialized, Role: FOLLOWER
I20260811 22:21:17.813512 16673 consensus_queue.cc:260] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.813946 16185 heartbeater.cc:503] Master 127.29.3.126:43269 was elected leader, sending a full tablet report...
I20260811 22:21:17.814107 16671 ts_tablet_manager.cc:1428] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:21:17.814518 16673 ts_tablet_manager.cc:1428] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 22:21:17.814833 16639 heartbeater.cc:503] Master 127.29.3.126:43269 was elected leader, sending a full tablet report...
I20260811 22:21:17.849932 16683 raft_consensus.cc:482] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:21:17.850195 16683 raft_consensus.cc:504] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.851456 16683 leader_election.cc:288] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 927f0df6047647f4a7c8f8867f37cf7b (127.29.3.66:39683), 1b905dacb3184bbf88bbb9d80609be06 (127.29.3.65:34803)
I20260811 22:21:17.855625 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.851721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39683 (local address 127.29.3.67:45605)
0811 22:21:17.851864 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.851881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.851988 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.852251 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.852255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.852280 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:21:17.852405 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.852412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.853304 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.853307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.854516 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.854528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.855258 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.855261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.855345 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.855367 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.855420 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.855477 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 22:21:17.855643 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.851817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:45605 (local address 127.29.3.66:39683)
0811 22:21:17.851944 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.851949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.851964 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:21:17.852080 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.852083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.852130 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.852197 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:21:17.852513 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.853153 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.854639 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.855141 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.855250 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.855316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.855447 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.855522 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:21:17.855963 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.852187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:46011 (local address 127.29.3.65:34803)
0811 22:21:17.852318 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.852322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.852515 (+ 193us) server_negotiation.cc:407] Connection header received
0811 22:21:17.852621 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.852624 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.852669 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.852760 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:21:17.853055 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.853604 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.854913 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.855425 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.855520 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.855575 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.855783 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.855866 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:21:17.856308 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.852001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34803 (local address 127.29.3.67:46011)
0811 22:21:17.852440 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.852452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:21:17.852537 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:21:17.852801 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:21:17.852803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:21:17.852827 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:21:17.852949 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.852956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.853725 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.853729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.854811 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:21:17.854819 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.855536 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.855540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:21:17.855633 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.855655 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:21:17.856113 (+ 458us) client_negotiation.cc:240] Negotiation successful
0811 22:21:17.856188 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":130,"threads_started":1}
I20260811 22:21:17.856323 16330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182432b6a2348e9ac65f94f72895fab" candidate_uuid: "490d3eb868704c83afbf32e33ac68beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927f0df6047647f4a7c8f8867f37cf7b" is_pre_election: true
I20260811 22:21:17.856654 16330 raft_consensus.cc:2454] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490d3eb868704c83afbf32e33ac68beb in term 0.
I20260811 22:21:17.856875 16115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182432b6a2348e9ac65f94f72895fab" candidate_uuid: "490d3eb868704c83afbf32e33ac68beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b905dacb3184bbf88bbb9d80609be06" is_pre_election: true
I20260811 22:21:17.857103 16453 leader_election.cc:302] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490d3eb868704c83afbf32e33ac68beb, 927f0df6047647f4a7c8f8867f37cf7b; no voters:
I20260811 22:21:17.857177 16115 raft_consensus.cc:2454] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 490d3eb868704c83afbf32e33ac68beb in term 0.
I20260811 22:21:17.857347 16683 raft_consensus.cc:2781] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:21:17.857430 16683 raft_consensus.cc:482] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:21:17.857470 16683 raft_consensus.cc:3037] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:21:17.858744 16683 raft_consensus.cc:504] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.859061 16683 leader_election.cc:288] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [CANDIDATE]: Term 1 election: Requested vote from peers 927f0df6047647f4a7c8f8867f37cf7b (127.29.3.66:39683), 1b905dacb3184bbf88bbb9d80609be06 (127.29.3.65:34803)
I20260811 22:21:17.859498 16115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182432b6a2348e9ac65f94f72895fab" candidate_uuid: "490d3eb868704c83afbf32e33ac68beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b905dacb3184bbf88bbb9d80609be06"
I20260811 22:21:17.859501 16330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6182432b6a2348e9ac65f94f72895fab" candidate_uuid: "490d3eb868704c83afbf32e33ac68beb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "927f0df6047647f4a7c8f8867f37cf7b"
I20260811 22:21:17.859637 16330 raft_consensus.cc:3037] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:21:17.859628 16115 raft_consensus.cc:3037] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:21:17.860888 16330 raft_consensus.cc:2454] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490d3eb868704c83afbf32e33ac68beb in term 1.
I20260811 22:21:17.860888 16115 raft_consensus.cc:2454] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 490d3eb868704c83afbf32e33ac68beb in term 1.
I20260811 22:21:17.861151 16453 leader_election.cc:302] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 490d3eb868704c83afbf32e33ac68beb, 927f0df6047647f4a7c8f8867f37cf7b; no voters:
I20260811 22:21:17.861307 16683 raft_consensus.cc:2781] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:21:17.861552 16683 raft_consensus.cc:686] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 1 LEADER]: Becoming Leader. State: Replica: 490d3eb868704c83afbf32e33ac68beb, State: Running, Role: LEADER
I20260811 22:21:17.861860 16683 consensus_queue.cc:237] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } }
I20260811 22:21:17.865170 15844 catalog_manager.cc:5495] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb reported cstate change: term changed from 0 to 1, leader changed from <none> to 490d3eb868704c83afbf32e33ac68beb (127.29.3.67). New cstate: current_term: 1 leader_uuid: "490d3eb868704c83afbf32e33ac68beb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "490d3eb868704c83afbf32e33ac68beb" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 40229 } health_report { overall_health: HEALTHY } } }
I20260811 22:21:17.885401 16668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.882314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.29.3.67:40229)
0811 22:21:17.882421 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 22:21:17.882424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:21:17.882438 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:21:17.882548 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:21:17.882551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:21:17.882623 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:21:17.882714 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:21:17.882909 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.883403 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.884165 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:21:17.884365 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:21:17.884429 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:21:17.884659 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:21:17.885081 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:21:17.885182 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 22:21:17.885205 (+ 23us) server_negotiation.cc:299] Negotiation successful
0811 22:21:17.885267 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 22:21:17.891498 16115 raft_consensus.cc:1261] T 6182432b6a2348e9ac65f94f72895fab P 1b905dacb3184bbf88bbb9d80609be06 [term 1 FOLLOWER]: Refusing update from remote peer 490d3eb868704c83afbf32e33ac68beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:21:17.891498 16330 raft_consensus.cc:1261] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 1 FOLLOWER]: Refusing update from remote peer 490d3eb868704c83afbf32e33ac68beb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:21:17.892297 16685 consensus_queue.cc:1035] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "927f0df6047647f4a7c8f8867f37cf7b" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:21:17.892488 16683 consensus_queue.cc:1035] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b905dacb3184bbf88bbb9d80609be06" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:21:17.900539 16690 mvcc.cc:205] Tried to move back new op lower bound from 7317450251835744256 to 7317450251725103104. Current Snapshot: MvccSnapshot[applied={T|T < 7317450251835744256}]
I20260811 22:21:17.901757 16691 mvcc.cc:205] Tried to move back new op lower bound from 7317450251835744256 to 7317450251725103104. Current Snapshot: MvccSnapshot[applied={T|T < 7317450251835744256}]
I20260811 22:21:17.908484 15844 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:21:17.908653 15844 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:21:17.911129 15844 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 891899724c78456c9984ef15c3636340: Sending DeleteTablet for 3 replicas of tablet 6182432b6a2348e9ac65f94f72895fab
I20260811 22:21:17.911978 16528 tablet_service.cc:1514] Processing DeleteTablet for tablet 6182432b6a2348e9ac65f94f72895fab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:21:17 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:49190
I20260811 22:21:17.911991 16079 tablet_service.cc:1514] Processing DeleteTablet for tablet 6182432b6a2348e9ac65f94f72895fab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:21:17 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:36402
I20260811 22:21:17.912122 16298 tablet_service.cc:1514] Processing DeleteTablet for tablet 6182432b6a2348e9ac65f94f72895fab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:21:17 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:48868
I20260811 22:21:17.912528 16703 tablet_replica.cc:330] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: stopping tablet replica
I20260811 22:21:17.912868 16703 raft_consensus.cc:2229] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:21:17.913185 16701 tablet_replica.cc:330] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: stopping tablet replica
I20260811 22:21:17.913323 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 15960
I20260811 22:21:17.913223 16703 raft_consensus.cc:2258] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:21:17.922576 16701 raft_consensus.cc:2229] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:21:17.923179 16701 raft_consensus.cc:2258] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:21:17.923357 16703 ts_tablet_manager.cc:1905] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:21:17.924695 16701 ts_tablet_manager.cc:1905] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:21:17.926800 16703 ts_tablet_manager.cc:1918] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:21:17.926918 16703 log.cc:1196] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/6182432b6a2348e9ac65f94f72895fab
I20260811 22:21:17.927265 16703 ts_tablet_manager.cc:1939] T 6182432b6a2348e9ac65f94f72895fab P 927f0df6047647f4a7c8f8867f37cf7b: Deleting consensus metadata
I20260811 22:21:17.927714 16701 ts_tablet_manager.cc:1918] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:21:17.927870 16701 log.cc:1196] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/6182432b6a2348e9ac65f94f72895fab
I20260811 22:21:17.928256 16701 ts_tablet_manager.cc:1939] T 6182432b6a2348e9ac65f94f72895fab P 490d3eb868704c83afbf32e33ac68beb: Deleting consensus metadata
I20260811 22:21:17.928741 15821 catalog_manager.cc:4841] TS 927f0df6047647f4a7c8f8867f37cf7b (127.29.3.66:39683): tablet 6182432b6a2348e9ac65f94f72895fab (table test-table [id=a561a6f02cd2403887527ddcf0808593]) successfully deleted
I20260811 22:21:17.929481 15815 catalog_manager.cc:4841] TS 490d3eb868704c83afbf32e33ac68beb (127.29.3.67:40229): tablet 6182432b6a2348e9ac65f94f72895fab (table test-table [id=a561a6f02cd2403887527ddcf0808593]) successfully deleted
W20260811 22:21:17.933081 15821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:34803 (error 108)
I20260811 22:21:17.934001 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 16191
I20260811 22:21:17.934108 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:21:17.933836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34803 (local address 127.0.0.1:36406)
0811 22:21:17.933958 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0811 22:21:17.934024 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 22:21:17.934639 15821 catalog_manager.cc:4584] TS 1b905dacb3184bbf88bbb9d80609be06 (127.29.3.65:34803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6182432b6a2348e9ac65f94f72895fab: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34803: connect: Connection refused (error 111)
I20260811 22:21:17.946413 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 16415
I20260811 22:21:17.966075 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 15784
2026-08-11T22:21:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2441 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:21:18 dist-test-slave-4tdj krb5kdc[16708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:21:18 dist-test-slave-4tdj krb5kdc[16708](info): set up 2 sockets
Aug 11 22:21:18 dist-test-slave-4tdj krb5kdc[16708](info): commencing operation
krb5kdc: starting...
W20260811 22:21:19.154795 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:21:19 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:21:19Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/rangeradmin_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/rangeradmin_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/rangerlookup_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/rangerlookup_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
I20260811 22:21:19.322824 29709 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCdqKk8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/postgres -l logfile start
2026-08-11 22:21:21.694 UTC [16767] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 22:21:21.695 UTC [16767] LOG: listening on IPv4 address "127.29.3.84", port 48291
2026-08-11 22:21:21.698 UTC [16767] LOG: listening on Unix socket "/tmp/.s.PGSQL.48291"
2026-08-11 22:21:21.727 UTC [16770] LOG: database system was shut down at 2026-08-11 22:21:20 UTC
2026-08-11 22:21:21.731 UTC [16767] LOG: database system is ready to accept connections
I20260811 22:21:22.824599 29709 mini_postgres.cc:96] Postgres bound to 48291
2026-08-11 22:21:22.832 UTC [16786] FATAL: database "slave" does not exist
127.29.3.84:48291 - accepting connections
I20260811 22:21:22.832911 29709 mini_ranger.cc:161] Starting Ranger...
I20260811 22:21:22.849689 29709 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 22:21:23.288317 29709 mini_ranger.cc:87] Created ranger Postgres database
I20260811 22:21:23.288400 29709 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-admin
2026-08-11 22:21:23,563 [I] DB FLAVOR :POSTGRES
2026-08-11 22:21:23,563 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:21:23,564 [I] Checking connection..
2026-08-11 22:21:23,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 22:21:23,815 [I] Checking connection passed.
2026-08-11 22:21:23,815 [I] --------- Verifying version history table ---------
2026-08-11 22:21:23,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:21:24,077 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:21:24,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:21:24,317 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:21:24,317 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 22:21:24,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 22:21:24.572 UTC [16890] WARNING: there is no transaction in progress
2026-08-11 22:21:24,581 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 22:21:24,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:21:24,844 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 22:21:24,844 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:21:24,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 22:21:25,113 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 22:21:25,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4tdj.c.gcp-upstream.internal','N') ;"
2026-08-11 22:21:25,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:21:25,897 [I] Table x_portal_user does not exist in database ranger
2026-08-11 22:21:25,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:21:26,169 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 22:21:26,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:21:26,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:21:26,684 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 22:21:26,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 22:21:27.603 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.619 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.634 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.649 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.667 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.685 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.749 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.756 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.770 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.781 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.790 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.804 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.814 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.820 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.826 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:27.832 UTC [17079] WARNING: there is no transaction in progress
2026-08-11 22:21:28,475 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 22:21:28,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 22:21:28,733 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 22:21:28,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:21:28,999 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 22:21:28,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:21:29,257 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 22:21:29,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:21:29,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:21:29,762 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4tdj.c.gcp-upstream.internal';"
2026-08-11 22:21:30,020 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 22:21:30,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 22:21:30,279 [I] DB_PATCHES have already been applied
I20260811 22:21:30.285385 29709 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCdqKk8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 22:21:30.285454 29709 mini_ranger.cc:193] Using host: 127.29.3.84
I20260811 22:21:30.289791 29709 mini_ranger.cc:227] Ranger admin URL: http://127.29.3.84:53253
Aug 11, 2026 10:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 10:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 10:21:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 10:21:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 10:21:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 22:21:31.522655 29709 mini_ranger.cc:160] Time spent starting Ranger: real 8.690s user 0.000s sys 0.013s
Aug 11, 2026 10:21:52 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 10:21:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 11, 2026 10:21:52 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 10:21:52 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 10:21:53 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 22:21:53.983619 29709 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/rangerkms_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/rangerkms_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.3.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.3.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 22:21:54 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486914, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 22:21:54.075377 29709 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 22:21:54.086266 29709 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 22:21:54.893975 29709 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 22:21:54.894062 29709 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms
I20260811 22:21:54.894191 29709 mini_ranger_kms.cc:227] Using postgres at 127.29.3.84:48291
2026-08-11 22:21:55,285 [I] DB FLAVOR :POSTGRES
2026-08-11 22:21:55,285 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:21:55,285 [I] Checking connection
2026-08-11 22:21:55,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 22:21:55,552 [I] connection success
2026-08-11 22:21:55,552 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 22:21:55,552 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 22:21:55,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 22:21:55,796 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 22:21:55,796 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:21:55,796 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 22:21:55,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:48291/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 22:21:56,068 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 22:21:56.563484 29709 mini_ranger_kms.cc:310] Created kms service
I20260811 22:21:56.771401 29709 mini_ranger_kms.cc:326] Created kudu user
I20260811 22:21:56.813791 29709 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 22:21:56 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486914, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:21:57.240527 29709 mini_ranger_kms.cc:384] Added ranger policy
I20260811 22:21:57.240696 29709 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 22:21:57.240728 29709 mini_ranger_kms.cc:240] Using host: 127.29.3.84
I20260811 22:21:57.246430 29709 mini_ranger_kms.cc:276] Ranger KMS PID: 17407
I20260811 22:21:57.246555 29709 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.3.84:49795
Aug 11, 2026 10:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 10:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 10:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 10:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 22:21:59 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786486919, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.3.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:21:59.871820 29709 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.796s user 0.000s sys 0.365s
I20260811 22:21:59.871929 29709 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 22:21:59.872030 29709 mini_ranger_kms.cc:401] 127.29.3.84:49795/kms/v1/keys
Aug 11 22:22:00 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486919, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.3.84@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
Aug 11 22:22:00 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486919, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.3.84@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
Aug 11 22:22:03 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:22:03.335788 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:34847
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:34847
--ranger_config_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:49795/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:03.486689 17504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:03.486981 17504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:03.487021 17504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:03.491457 17504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:22:03.491567 17504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:03.491587 17504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:03.491603 17504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:22:03.491619 17504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:22:03.498268 17504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.3.84:49795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:34847
--ranger_config_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:34847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:03.499467 17504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:03.500554 17504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:03.510700 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:03.510942 17504 server_base.cc:1034] running on GCE node
W20260811 22:22:03.510763 17510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:03.510772 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:03.511807 17504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:03.512997 17504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:03.514187 17504 hybrid_clock.cc:648] HybridClock initialized: now 1786486923514168 us; error 30 us; skew 500 ppm
Aug 11 22:22:03 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486923, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:03.520561 17504 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:22:03.521664 17504 webserver.cc:460] Webserver started at http://127.29.3.126:40963/ using document root <none> and password file <none>
I20260811 22:22:03.522257 17504 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:03.522351 17504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:03.522578 17504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:03 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486923, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:03.581943 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "d51994728b2d4d77a89e1f4d82ec4907"
format_stamp: "Formatted at 2026-08-11 22:22:03 on dist-test-slave-4tdj"
server_key: "a46de38486a78cf6293c076bfa6198a6"
server_key_iv: "a3e0a2487bd52ce5e0c25864f1975960"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:03.582703 17504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "d51994728b2d4d77a89e1f4d82ec4907"
format_stamp: "Formatted at 2026-08-11 22:22:03 on dist-test-slave-4tdj"
server_key: "a46de38486a78cf6293c076bfa6198a6"
server_key_iv: "a3e0a2487bd52ce5e0c25864f1975960"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:03.587445 17504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:22:03.606979 17532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:03.608414 17504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:22:03.608568 17504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "d51994728b2d4d77a89e1f4d82ec4907"
format_stamp: "Formatted at 2026-08-11 22:22:03 on dist-test-slave-4tdj"
server_key: "a46de38486a78cf6293c076bfa6198a6"
server_key_iv: "a3e0a2487bd52ce5e0c25864f1975960"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:03.608716 17504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:03.634109 17504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:03.637562 17504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:03.637840 17504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:03.652256 17504 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:34847
I20260811 22:22:03.652328 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:34847 every 8 connection(s)
I20260811 22:22:03.653391 17504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:22:03.655756 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 17504
I20260811 22:22:03.655891 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:22:03.656224 29709 external_mini_cluster.cc:1428] Setting key 8e47c9aeac8da6dc03162d41d04bb28c
I20260811 22:22:03.656843 17623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:03.665442 17623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907: Bootstrap starting.
Aug 11 22:22:03 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:03.668502 17623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:03.670210 17623 log.cc:824] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:03.672777 17623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907: No bootstrap required, opened a new log
I20260811 22:22:03.673923 17634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:03.657834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.29.3.126:34847)
0811 22:22:03.658442 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:03.658474 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:03.658517 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:22:03.659442 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:03.659467 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:03.659910 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:03.660309 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:03.661265 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:03.663050 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:03.664277 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:03.665040 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:03.665211 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:03.668635 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:03.668648 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:03.668669 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:03.668706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:03.670973 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:03.671443 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:03.671448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:03.671452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:03.671562 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:03.671833 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:03.671836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:03.671838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:03.672258 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:03.672428 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:03.672727 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 22:22:03.673164 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":215,"threads_started":1}
I20260811 22:22:03.677878 17623 raft_consensus.cc:348] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51994728b2d4d77a89e1f4d82ec4907" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34847 } }
I20260811 22:22:03.678189 17623 raft_consensus.cc:374] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:03.678264 17623 raft_consensus.cc:729] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51994728b2d4d77a89e1f4d82ec4907, State: Initialized, Role: FOLLOWER
I20260811 22:22:03.678789 17623 consensus_queue.cc:260] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51994728b2d4d77a89e1f4d82ec4907" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34847 } }
I20260811 22:22:03.678938 17623 raft_consensus.cc:388] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:22:03.678996 17623 raft_consensus.cc:482] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:22:03.679087 17623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:03.680603 17623 raft_consensus.cc:504] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51994728b2d4d77a89e1f4d82ec4907" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34847 } }
I20260811 22:22:03.681025 17623 leader_election.cc:302] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d51994728b2d4d77a89e1f4d82ec4907; no voters:
I20260811 22:22:03.681444 17623 leader_election.cc:288] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:22:03.681636 17640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:22:03.682183 17640 raft_consensus.cc:686] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [term 1 LEADER]: Becoming Leader. State: Replica: d51994728b2d4d77a89e1f4d82ec4907, State: Running, Role: LEADER
I20260811 22:22:03.682598 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51994728b2d4d77a89e1f4d82ec4907" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34847 } }
I20260811 22:22:03.682963 17623 sys_catalog.cc:564] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:22:03.684545 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d51994728b2d4d77a89e1f4d82ec4907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51994728b2d4d77a89e1f4d82ec4907" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34847 } } }
I20260811 22:22:03.684679 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [sys.catalog]: This master's current role is: LEADER
I20260811 22:22:03.685134 17642 sys_catalog.cc:455] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d51994728b2d4d77a89e1f4d82ec4907. Latest consensus state: current_term: 1 leader_uuid: "d51994728b2d4d77a89e1f4d82ec4907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d51994728b2d4d77a89e1f4d82ec4907" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 34847 } } }
I20260811 22:22:03.685248 17658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:22:03.685313 17642 sys_catalog.cc:458] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907 [sys.catalog]: This master's current role is: LEADER
I20260811 22:22:03.689777 17658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:22:03.690339 17623 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 22:22:03.710544 17658 catalog_manager.cc:1347] Generated new cluster ID: f0945ff42bbb458ebb8f32987b59810c
I20260811 22:22:03.710639 17658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:22:03.747056 17658 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:22:03.748600 17658 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:22:03.758464 17658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907: Generated new TSK 0
I20260811 22:22:03.759565 17658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:22:04.621867 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:49795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34847
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:04.803290 17714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:04.803572 17714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:04.803637 17714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:04.807739 17714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:04.807803 17714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:04.807881 17714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:22:04.814134 17714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.3.84:49795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:04.815568 17714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:04.817042 17714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 22:22:04.828171 17714 server_base.cc:1034] running on GCE node
W20260811 22:22:04.828119 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:04.828119 17730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:04.828454 17728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:04.828918 17714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:04.829568 17714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:04.830725 17714 hybrid_clock.cc:648] HybridClock initialized: now 1786486924830692 us; error 51 us; skew 500 ppm
Aug 11 22:22:04 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486924, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:04.834107 17714 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:22:04.835085 17714 webserver.cc:460] Webserver started at http://127.29.3.65:43245/ using document root <none> and password file <none>
I20260811 22:22:04.835681 17714 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:04.835747 17714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:04.835973 17714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:04 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486924, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:04.849614 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "c38a3944b4c74fc7a63617d5d9ffb849"
format_stamp: "Formatted at 2026-08-11 22:22:04 on dist-test-slave-4tdj"
server_key: "30879d68bafae2620da6d04adb31d442"
server_key_iv: "968843e19bb0e79f07f604a6731c1697"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:04.850592 17714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "c38a3944b4c74fc7a63617d5d9ffb849"
format_stamp: "Formatted at 2026-08-11 22:22:04 on dist-test-slave-4tdj"
server_key: "30879d68bafae2620da6d04adb31d442"
server_key_iv: "968843e19bb0e79f07f604a6731c1697"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:04.856076 17714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 22:22:04.874665 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:04.876564 17714 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260811 22:22:04.876727 17714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "c38a3944b4c74fc7a63617d5d9ffb849"
format_stamp: "Formatted at 2026-08-11 22:22:04 on dist-test-slave-4tdj"
server_key: "30879d68bafae2620da6d04adb31d442"
server_key_iv: "968843e19bb0e79f07f604a6731c1697"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:04.876895 17714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:04.901641 17714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:04.906420 17714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:04.906679 17714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:04.907486 17714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:04.908670 17714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:04.908771 17714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:04.908854 17714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:04.908901 17714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:04.934860 17714 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:34353
I20260811 22:22:04.936321 17714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:22:04.937193 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 17714
I20260811 22:22:04.937325 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:22:04.937642 29709 external_mini_cluster.cc:1428] Setting key 1aadb74290d0c848278cfa60f11bfe68
I20260811 22:22:04.937090 17950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:34353 every 8 connection(s)
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Aug 11 22:22:04 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486924, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:04.966523 17967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:04.948986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:34847 (local address 127.29.3.65:60707)
0811 22:22:04.951110 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:04.951175 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:04.952453 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:04.953380 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:04.953388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:04.953763 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:04.954359 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:04.954381 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:04.955530 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:04.955533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:04.956849 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:04.956862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:04.957633 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:04.957638 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:04.957845 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:04.958700 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:04.958723 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:04.960480 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:04.962966 (+ 2486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:04.962973 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:04.962993 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:04.963384 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:04.963738 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:04.963741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:04.963743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:04.963942 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:04.964444 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:04.964451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:04.964826 (+ 375us) client_negotiation.cc:769] Sending connection context
0811 22:22:04.965030 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 22:22:04.965401 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1853,"thread_start_us":239,"threads_started":1}
I20260811 22:22:04.967171 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:04.950406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:60707 (local address 127.29.3.126:34847)
0811 22:22:04.950758 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:04.950763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:04.952810 (+ 2047us) server_negotiation.cc:407] Connection header received
0811 22:22:04.952873 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:04.952876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:04.952939 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:04.953028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:04.954580 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:04.955360 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:04.956975 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:04.957317 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:04.957426 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:04.960730 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:04.960734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:04.960737 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:04.960771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:04.962814 (+ 2043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:04.963507 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:04.963511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:04.963513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:04.963625 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:04.964056 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:04.964060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:04.964061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:04.964301 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:04.964387 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:04.966859 (+ 2472us) server_negotiation.cc:299] Negotiation successful
0811 22:22:04.967002 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":92,"threads_started":1}
I20260811 22:22:04.968799 17952 heartbeater.cc:347] Connected to a master server at 127.29.3.126:34847
I20260811 22:22:04.969092 17952 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:04.969640 17952 heartbeater.cc:511] Master 127.29.3.126:34847 requested a full tablet report, sending...
I20260811 22:22:04.971485 17564 ts_manager.cc:194] Registered new tserver with Master: c38a3944b4c74fc7a63617d5d9ffb849 (127.29.3.65:34353)
I20260811 22:22:04.973294 17564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:60707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:22:04.978395 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:49795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34847
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 22:22:05.181921 17979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:05.182282 17979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:05.182389 17979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:05.188951 17979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:05.189041 17979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:05.189129 17979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:22:05.198108 17979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.3.84:49795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:05.199585 17979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:05.200938 17979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:05.215583 17991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:05.216145 17995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:05.216388 17979 server_base.cc:1034] running on GCE node
W20260811 22:22:05.216389 17990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:05.217087 17979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:05.217864 17979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:05.219051 17979 hybrid_clock.cc:648] HybridClock initialized: now 1786486925219026 us; error 44 us; skew 500 ppm
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:05.222008 17979 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:22:05.223065 17979 webserver.cc:460] Webserver started at http://127.29.3.66:43179/ using document root <none> and password file <none>
I20260811 22:22:05.223645 17979 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:05.223721 17979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:05.223958 17979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:05.248397 17979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "e1d867ffa60942929ef1b855b6b8dd75"
format_stamp: "Formatted at 2026-08-11 22:22:05 on dist-test-slave-4tdj"
server_key: "395a7789f4eeec0726d32d2884393b7c"
server_key_iv: "235afe819940261b1dcf8109068ed37b"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:05.249366 17979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "e1d867ffa60942929ef1b855b6b8dd75"
format_stamp: "Formatted at 2026-08-11 22:22:05 on dist-test-slave-4tdj"
server_key: "395a7789f4eeec0726d32d2884393b7c"
server_key_iv: "235afe819940261b1dcf8109068ed37b"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:05.259196 17979 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260811 22:22:05.287348 18007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:05.289134 17979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:22:05.289279 17979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "e1d867ffa60942929ef1b855b6b8dd75"
format_stamp: "Formatted at 2026-08-11 22:22:05 on dist-test-slave-4tdj"
server_key: "395a7789f4eeec0726d32d2884393b7c"
server_key_iv: "235afe819940261b1dcf8109068ed37b"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:05.289435 17979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:05.308233 17979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:05.313887 17979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:05.314189 17979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:05.315069 17979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:05.316453 17979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:05.316536 17979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:05.316602 17979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:05.316639 17979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:05.338315 17979 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:39393
I20260811 22:22:05.339671 17979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:22:05.344233 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 17979
I20260811 22:22:05.344398 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:22:05.344766 29709 external_mini_cluster.cc:1428] Setting key 13705da3dec4c62d0cf90702ae131156
I20260811 22:22:05.350575 18211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:39393 every 8 connection(s)
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:22:05.371707 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.342159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:48033 (local address 127.29.3.126:34847)
0811 22:22:05.342392 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:05.342397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:05.351238 (+ 8841us) server_negotiation.cc:407] Connection header received
0811 22:22:05.352501 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:05.352515 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:05.352578 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:05.352672 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:05.354695 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.355533 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.357432 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.357795 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.357891 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.362081 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:05.362086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:05.362090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:05.362125 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:05.364092 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:05.364812 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:05.364816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:05.364818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:05.364920 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:05.369808 (+ 4888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:05.369813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:05.369815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:05.370128 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:05.370236 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:05.371189 (+ 953us) server_negotiation.cc:299] Negotiation successful
0811 22:22:05.371449 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:22:05.373519 18217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.342730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:34847 (local address 127.29.3.66:48033)
0811 22:22:05.351041 (+ 8311us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:05.351114 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:05.352260 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:05.353018 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:05.353029 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:05.353545 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:05.354493 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.354528 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.355708 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.355711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.357280 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.357294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.358116 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.358121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.358423 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.359270 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:05.359292 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:05.361825 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:05.364263 (+ 2438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:05.364270 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:05.364288 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:05.364676 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:05.369392 (+ 4716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:05.369396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:05.369398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:05.369653 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:05.370344 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:05.370353 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:05.370763 (+ 410us) client_negotiation.cc:769] Sending connection context
0811 22:22:05.371936 (+ 1173us) client_negotiation.cc:240] Negotiation successful
0811 22:22:05.372363 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8030,"thread_start_us":307,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
I20260811 22:22:05.374984 18213 heartbeater.cc:347] Connected to a master server at 127.29.3.126:34847
I20260811 22:22:05.375255 18213 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:05.375903 18213 heartbeater.cc:511] Master 127.29.3.126:34847 requested a full tablet report, sending...
I20260811 22:22:05.377153 17564 ts_manager.cc:194] Registered new tserver with Master: e1d867ffa60942929ef1b855b6b8dd75 (127.29.3.66:39393)
I20260811 22:22:05.377995 17564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:48033
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:22:05.399065 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:49795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34847
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:43439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:05.588092 18223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:05.588687 18223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:05.588826 18223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:05.594725 18223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:05.594874 18223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:05.594998 18223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:22:05.603616 18223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:43439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.3.84:49795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:34847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:05.605283 18223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:05.606777 18223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:05.623431 18229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:05.623699 18230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:05.624594 18223 server_base.cc:1034] running on GCE node
W20260811 22:22:05.632509 18233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:05.633271 18223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:05.634155 18223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:05.637521 18223 hybrid_clock.cc:648] HybridClock initialized: now 1786486925637412 us; error 100 us; skew 500 ppm
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:05.643496 18223 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:22:05.644799 18223 webserver.cc:460] Webserver started at http://127.29.3.67:38617/ using document root <none> and password file <none>
I20260811 22:22:05.645597 18223 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:05.645740 18223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:05.646039 18223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:05.695734 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "b90e4fac45f548dd97f66d929ef1b4df"
format_stamp: "Formatted at 2026-08-11 22:22:05 on dist-test-slave-4tdj"
server_key: "b35b1fd95fb3c70c38735fcd29826f06"
server_key_iv: "fd5ef21eac5aac6cec6c23954f77d6ad"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:05.698158 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "b90e4fac45f548dd97f66d929ef1b4df"
format_stamp: "Formatted at 2026-08-11 22:22:05 on dist-test-slave-4tdj"
server_key: "b35b1fd95fb3c70c38735fcd29826f06"
server_key_iv: "fd5ef21eac5aac6cec6c23954f77d6ad"
server_key_version: "kuduclusterkey@0"
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:05.722193 18223 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.004s sys 0.000s
I20260811 22:22:05.758725 18245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:05.761006 18223 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.000s
I20260811 22:22:05.761150 18223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "b90e4fac45f548dd97f66d929ef1b4df"
format_stamp: "Formatted at 2026-08-11 22:22:05 on dist-test-slave-4tdj"
server_key: "b35b1fd95fb3c70c38735fcd29826f06"
server_key_iv: "fd5ef21eac5aac6cec6c23954f77d6ad"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:05.761305 18223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:05.809248 18223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:05.813189 18223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:05.813572 18223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:05.814419 18223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:05.815819 18223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:05.815924 18223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:05.816022 18223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:05.816082 18223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:05.844411 18223 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:34521
I20260811 22:22:05.845747 18223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:22:05.847877 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 18223
I20260811 22:22:05.848007 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:22:05.848285 29709 external_mini_cluster.cc:1428] Setting key 997135f37599ed26125975e703a8452c
I20260811 22:22:05.848485 18447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:34521 every 8 connection(s)
Aug 11 22:22:05 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:05.868501 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.850488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:49917 (local address 127.29.3.126:34847)
0811 22:22:05.850701 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:05.850705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:05.853065 (+ 2360us) server_negotiation.cc:407] Connection header received
0811 22:22:05.854575 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:05.854580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:05.854648 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:05.854700 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:05.856539 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.857390 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.859202 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.859536 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.859630 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.864170 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:05.864174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:05.864176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:05.864207 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:05.865842 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:05.866520 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:05.866523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:05.866525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:05.866625 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:05.867061 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:05.867064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:05.867066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:05.867290 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:05.867366 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:05.868215 (+ 849us) server_negotiation.cc:299] Negotiation successful
0811 22:22:05.868342 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260811 22:22:05.870671 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.851493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:34847 (local address 127.29.3.67:49917)
0811 22:22:05.852812 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:05.852873 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:05.854274 (+ 1401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:05.855138 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:05.855147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:05.855572 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:05.856353 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.856374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.857559 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.857562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.859055 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.859067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.859855 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.859859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.860090 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.860857 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:05.860879 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:05.863971 (+ 3092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:05.865989 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:05.865996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:05.866017 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:05.866404 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:05.866763 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:05.866766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:05.866768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:05.866957 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:05.867434 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:05.867441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:05.867934 (+ 493us) client_negotiation.cc:769] Sending connection context
0811 22:22:05.869191 (+ 1257us) client_negotiation.cc:240] Negotiation successful
0811 22:22:05.869554 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":154,"threads_started":1}
I20260811 22:22:05.872388 18451 heartbeater.cc:347] Connected to a master server at 127.29.3.126:34847
I20260811 22:22:05.872655 18451 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:05.873237 18451 heartbeater.cc:511] Master 127.29.3.126:34847 requested a full tablet report, sending...
I20260811 22:22:05.874506 17564 ts_manager.cc:194] Registered new tserver with Master: b90e4fac45f548dd97f66d929ef1b4df (127.29.3.67:34521)
I20260811 22:22:05.875254 17564 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:49917
I20260811 22:22:05.882031 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:22:05.895356 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.884878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.29.3.126:34847)
0811 22:22:05.885006 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:05.885011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:05.885719 (+ 708us) server_negotiation.cc:407] Connection header received
0811 22:22:05.885917 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:05.885920 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:05.885970 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:05.886014 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:05.887115 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.887940 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.889130 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.889433 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.889508 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.891457 (+ 1949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:05.891461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:05.891464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:05.891494 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:05.893181 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:05.893694 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:05.893698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:05.893700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:05.893790 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:05.894161 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:05.894164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:05.894166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:05.894458 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:05.894579 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:05.894979 (+ 400us) server_negotiation.cc:299] Negotiation successful
0811 22:22:05.895150 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:22:05.900131 17564 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:22:05.905397 17564 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:22:05.948441 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.921484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.29.3.67:34521)
0811 22:22:05.936455 (+ 14971us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:05.936493 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:05.937066 (+ 573us) server_negotiation.cc:407] Connection header received
0811 22:22:05.937167 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:05.937174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:05.937432 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:05.937653 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:05.938734 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.940797 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.943058 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.943974 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.944077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.944231 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:05.948124 (+ 3893us) server_negotiation.cc:299] Negotiation successful
0811 22:22:05.948255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14822,"thread_start_us":188,"threads_started":1}
I20260811 22:22:05.948822 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.920934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:34521 (local address 127.0.0.1:36910)
0811 22:22:05.936863 (+ 15929us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:05.936877 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:05.936966 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:05.938393 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:05.938400 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:05.938424 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:05.938594 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.938600 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.941279 (+ 2679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.941283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.942845 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.942860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.946395 (+ 3535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.946399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.946549 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.946606 (+ 57us) client_negotiation.cc:769] Sending connection context
0811 22:22:05.946717 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 22:22:05.946864 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15858,"thread_start_us":186,"threads_started":1}
I20260811 22:22:05.952685 18329 tablet_service.cc:1467] Processing CreateTablet for tablet 04051e10661446c49f875b50f5c94e72 (DEFAULT_TABLE table=test-table [id=5c6612430b2a4d7ba584290a97c0b883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:22:05.953917 18329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04051e10661446c49f875b50f5c94e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:05.955206 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.920150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39393 (local address 127.0.0.1:33874)
0811 22:22:05.926588 (+ 6438us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:05.926607 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:05.926806 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:05.950368 (+ 23562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:05.950371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:05.950393 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:05.950491 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.950498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.951911 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.951916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.953228 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.953241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.954792 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.954796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.954897 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.954928 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:22:05.954978 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 22:22:05.955052 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6343,"thread_start_us":190,"threads_started":1}
I20260811 22:22:05.956195 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.920881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33874 (local address 127.29.3.66:39393)
0811 22:22:05.937972 (+ 17091us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:05.937979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:05.938002 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:22:05.938077 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:05.938083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:05.938266 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:05.941075 (+ 2809us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:05.950679 (+ 9604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.951764 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.953483 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.954642 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.955695 (+ 1053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.955859 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:05.955914 (+ 55us) server_negotiation.cc:299] Negotiation successful
0811 22:22:05.956007 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16972,"thread_start_us":111,"threads_started":1}
I20260811 22:22:05.966436 18072 tablet_service.cc:1467] Processing CreateTablet for tablet 04051e10661446c49f875b50f5c94e72 (DEFAULT_TABLE table=test-table [id=5c6612430b2a4d7ba584290a97c0b883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:22:05.969106 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04051e10661446c49f875b50f5c94e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:05.971563 18483 tablet_bootstrap.cc:492] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Bootstrap starting.
I20260811 22:22:05.972666 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.919251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.29.3.65:34353)
0811 22:22:05.944434 (+ 25183us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:05.944441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:05.944464 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:22:05.944574 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:05.944580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:05.944786 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:05.945345 (+ 559us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:05.945500 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.950521 (+ 5021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.957877 (+ 7356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.958786 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.969634 (+ 10848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.971072 (+ 1438us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:05.972259 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0811 22:22:05.972417 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25068,"thread_start_us":153,"threads_started":1}
I20260811 22:22:05.973361 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:05.917937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34353 (local address 127.0.0.1:53254)
0811 22:22:05.922189 (+ 4252us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:05.922252 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:05.936597 (+ 14345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:05.945044 (+ 8447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:05.945048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:05.945069 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:05.945169 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.945175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.956459 (+ 11284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.956463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.957699 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:05.957712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:05.971797 (+ 14085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:05.971802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:05.971932 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:05.971961 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 22:22:05.973093 (+ 1132us) client_negotiation.cc:240] Negotiation successful
0811 22:22:05.973206 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4037,"thread_start_us":484,"threads_started":1}
I20260811 22:22:05.975517 18483 tablet_bootstrap.cc:654] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:05.976425 17952 heartbeater.cc:503] Master 127.29.3.126:34847 was elected leader, sending a full tablet report...
I20260811 22:22:05.977208 17801 tablet_service.cc:1467] Processing CreateTablet for tablet 04051e10661446c49f875b50f5c94e72 (DEFAULT_TABLE table=test-table [id=5c6612430b2a4d7ba584290a97c0b883]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:22:05.978667 17801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04051e10661446c49f875b50f5c94e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:05.978642 18483 log.cc:824] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:05.992818 18488 tablet_bootstrap.cc:492] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Bootstrap starting.
I20260811 22:22:05.995754 18483 tablet_bootstrap.cc:492] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: No bootstrap required, opened a new log
I20260811 22:22:05.995998 18483 ts_tablet_manager.cc:1397] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.003s
I20260811 22:22:05.997717 18488 tablet_bootstrap.cc:654] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:05.999310 18488 log.cc:824] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:06.001976 18488 tablet_bootstrap.cc:492] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: No bootstrap required, opened a new log
I20260811 22:22:06.001876 18483 raft_consensus.cc:348] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.002177 18483 raft_consensus.cc:374] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:06.002197 18488 ts_tablet_manager.cc:1397] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260811 22:22:06.002239 18483 raft_consensus.cc:729] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b90e4fac45f548dd97f66d929ef1b4df, State: Initialized, Role: FOLLOWER
I20260811 22:22:06.002859 18483 consensus_queue.cc:260] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.004001 18483 ts_tablet_manager.cc:1428] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260811 22:22:06.004420 18451 heartbeater.cc:503] Master 127.29.3.126:34847 was elected leader, sending a full tablet report...
I20260811 22:22:06.011744 18489 tablet_bootstrap.cc:492] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: Bootstrap starting.
I20260811 22:22:06.011901 18488 raft_consensus.cc:348] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.012267 18488 raft_consensus.cc:374] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:06.012344 18488 raft_consensus.cc:729] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d867ffa60942929ef1b855b6b8dd75, State: Initialized, Role: FOLLOWER
I20260811 22:22:06.013113 18488 consensus_queue.cc:260] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.014616 18488 ts_tablet_manager.cc:1428] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20260811 22:22:06.015022 18213 heartbeater.cc:503] Master 127.29.3.126:34847 was elected leader, sending a full tablet report...
I20260811 22:22:06.016999 18489 tablet_bootstrap.cc:654] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:06.019127 18489 log.cc:824] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:06.021996 18489 tablet_bootstrap.cc:492] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: No bootstrap required, opened a new log
I20260811 22:22:06.022244 18489 ts_tablet_manager.cc:1397] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260811 22:22:06.029829 18489 raft_consensus.cc:348] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.030176 18489 raft_consensus.cc:374] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:06.030231 18489 raft_consensus.cc:729] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c38a3944b4c74fc7a63617d5d9ffb849, State: Initialized, Role: FOLLOWER
I20260811 22:22:06.030848 18489 consensus_queue.cc:260] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.032109 18489 ts_tablet_manager.cc:1428] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
W20260811 22:22:06.091966 18215 tablet.cc:2369] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:22:06.101019 18453 tablet.cc:2369] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 22:22:06 dist-test-slave-4tdj krb5kdc[16708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486925, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
W20260811 22:22:06.201143 17954 tablet.cc:2369] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:22:06.243598 18496 raft_consensus.cc:482] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:22:06.243891 18496 raft_consensus.cc:504] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.245333 18496 leader_election.cc:288] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1d867ffa60942929ef1b855b6b8dd75 (127.29.3.66:39393), b90e4fac45f548dd97f66d929ef1b4df (127.29.3.67:34521)
I20260811 22:22:06.263850 18478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:06.246257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:39235 (local address 127.29.3.66:39393)
0811 22:22:06.246482 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:06.246487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:06.250543 (+ 4056us) server_negotiation.cc:407] Connection header received
0811 22:22:06.254582 (+ 4039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:06.254587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:06.254646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:06.254699 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:06.255424 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.256215 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.262443 (+ 6228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.263142 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.263227 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:06.263289 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:06.263617 (+ 328us) server_negotiation.cc:299] Negotiation successful
0811 22:22:06.263714 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260811 22:22:06.263968 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:06.245609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39393 (local address 127.29.3.65:39235)
0811 22:22:06.250239 (+ 4630us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:06.250260 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:06.254403 (+ 4143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:06.254862 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:06.254865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:06.254887 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:06.255268 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:06.255280 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.256371 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.256374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:06.258068 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:06.258081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.263380 (+ 5299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.263383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:06.263480 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:06.263499 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:22:06.263563 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:22:06.263655 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4510,"thread_start_us":147,"threads_started":1}
I20260811 22:22:06.265051 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04051e10661446c49f875b50f5c94e72" candidate_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d867ffa60942929ef1b855b6b8dd75" is_pre_election: true
I20260811 22:22:06.265465 18112 raft_consensus.cc:2454] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c38a3944b4c74fc7a63617d5d9ffb849 in term 0.
I20260811 22:22:06.266103 17754 leader_election.cc:302] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c38a3944b4c74fc7a63617d5d9ffb849, e1d867ffa60942929ef1b855b6b8dd75; no voters:
I20260811 22:22:06.266604 18496 raft_consensus.cc:2781] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:22:06.266737 18496 raft_consensus.cc:482] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:22:06.266824 18496 raft_consensus.cc:3037] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:06.268589 18480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:06.246337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:34523 (local address 127.29.3.67:34521)
0811 22:22:06.246461 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:06.246466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:06.252052 (+ 5586us) server_negotiation.cc:407] Connection header received
0811 22:22:06.262464 (+ 10412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:06.262469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:06.262548 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:06.262604 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:06.263195 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.263959 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.267141 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.267796 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.267877 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:06.267945 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:06.268365 (+ 420us) server_negotiation.cc:299] Negotiation successful
0811 22:22:06.268456 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:22:06.269048 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:06.245700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:34521 (local address 127.29.3.65:34523)
0811 22:22:06.251827 (+ 6127us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:06.251843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:06.252177 (+ 334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:06.262760 (+ 10583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:06.262763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:06.262787 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:06.263035 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:06.263044 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.264174 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.264177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:06.266968 (+ 2791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:06.266983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.268087 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.268090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:06.268221 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:06.268246 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:22:06.268777 (+ 531us) client_negotiation.cc:240] Negotiation successful
0811 22:22:06.268864 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6044,"thread_start_us":270,"threads_started":1}
I20260811 22:22:06.269217 18496 raft_consensus.cc:504] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.270138 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04051e10661446c49f875b50f5c94e72" candidate_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e4fac45f548dd97f66d929ef1b4df" is_pre_election: true
I20260811 22:22:06.270663 18363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04051e10661446c49f875b50f5c94e72" candidate_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b90e4fac45f548dd97f66d929ef1b4df"
I20260811 22:22:06.270903 18363 raft_consensus.cc:3037] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:06.272925 18363 raft_consensus.cc:2454] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38a3944b4c74fc7a63617d5d9ffb849 in term 1.
I20260811 22:22:06.274034 17756 leader_election.cc:302] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b90e4fac45f548dd97f66d929ef1b4df, c38a3944b4c74fc7a63617d5d9ffb849; no voters:
I20260811 22:22:06.274538 18364 raft_consensus.cc:2362] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c38a3944b4c74fc7a63617d5d9ffb849 in term 1. Re-sending same reply.
I20260811 22:22:06.275784 18112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04051e10661446c49f875b50f5c94e72" candidate_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1d867ffa60942929ef1b855b6b8dd75"
I20260811 22:22:06.275936 18112 raft_consensus.cc:3037] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:06.277784 18112 raft_consensus.cc:2454] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c38a3944b4c74fc7a63617d5d9ffb849 in term 1.
I20260811 22:22:06.278434 18496 leader_election.cc:288] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [CANDIDATE]: Term 1 election: Requested vote from peers e1d867ffa60942929ef1b855b6b8dd75 (127.29.3.66:39393), b90e4fac45f548dd97f66d929ef1b4df (127.29.3.67:34521)
I20260811 22:22:06.278622 18496 raft_consensus.cc:2781] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:22:06.278710 18496 raft_consensus.cc:686] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [term 1 LEADER]: Becoming Leader. State: Replica: c38a3944b4c74fc7a63617d5d9ffb849, State: Running, Role: LEADER
I20260811 22:22:06.279094 18496 consensus_queue.cc:237] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } }
I20260811 22:22:06.283169 17564 catalog_manager.cc:5495] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 reported cstate change: term changed from 0 to 1, leader changed from <none> to c38a3944b4c74fc7a63617d5d9ffb849 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c38a3944b4c74fc7a63617d5d9ffb849" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 } health_report { overall_health: UNKNOWN } } }
I20260811 22:22:06.308661 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:06.302866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53260 (local address 127.29.3.65:34353)
0811 22:22:06.303067 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:06.303071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:06.303085 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:22:06.303180 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:06.303183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:06.303255 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:06.303362 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:22:06.303777 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.304554 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.305974 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:06.306378 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:06.306487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:06.307056 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:22:06.307691 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:22:06.307818 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:06.308292 (+ 474us) server_negotiation.cc:299] Negotiation successful
0811 22:22:06.308364 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260811 22:22:06.323908 18112 raft_consensus.cc:1261] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 1 FOLLOWER]: Refusing update from remote peer c38a3944b4c74fc7a63617d5d9ffb849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:22:06.323908 18364 raft_consensus.cc:1261] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 1 FOLLOWER]: Refusing update from remote peer c38a3944b4c74fc7a63617d5d9ffb849: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:22:06.324832 18496 consensus_queue.cc:1035] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b90e4fac45f548dd97f66d929ef1b4df" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 34521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:22:06.325276 18504 consensus_queue.cc:1035] T 04051e10661446c49f875b50f5c94e72 P c38a3944b4c74fc7a63617d5d9ffb849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1d867ffa60942929ef1b855b6b8dd75" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:22:06.342947 18507 mvcc.cc:205] Tried to move back new op lower bound from 7317450450209796096 to 7317450450043527168. Current Snapshot: MvccSnapshot[applied={T|T < 7317450450209796096}]
I20260811 22:22:06.350706 18508 mvcc.cc:205] Tried to move back new op lower bound from 7317450450209796096 to 7317450450043527168. Current Snapshot: MvccSnapshot[applied={T|T < 7317450450209796096}]
I20260811 22:22:06.353742 18509 mvcc.cc:205] Tried to move back new op lower bound from 7317450450209796096 to 7317450450043527168. Current Snapshot: MvccSnapshot[applied={T|T < 7317450450209796096}]
I20260811 22:22:06.360383 17564 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:22:06.360582 17564 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:22:06.363229 17564 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d51994728b2d4d77a89e1f4d82ec4907: Sending DeleteTablet for 3 replicas of tablet 04051e10661446c49f875b50f5c94e72
I20260811 22:22:06.364169 18072 tablet_service.cc:1514] Processing DeleteTablet for tablet 04051e10661446c49f875b50f5c94e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:22:06 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:33874
I20260811 22:22:06.364203 17801 tablet_service.cc:1514] Processing DeleteTablet for tablet 04051e10661446c49f875b50f5c94e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:22:06 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:53254
I20260811 22:22:06.364831 18523 tablet_replica.cc:330] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: stopping tablet replica
I20260811 22:22:06.364887 18329 tablet_service.cc:1514] Processing DeleteTablet for tablet 04051e10661446c49f875b50f5c94e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:22:06 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:36910
I20260811 22:22:06.365319 18523 raft_consensus.cc:2229] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:22:06.365708 18523 raft_consensus.cc:2258] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:22:06.365906 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 17714
I20260811 22:22:06.370482 18523 ts_tablet_manager.cc:1905] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:22:06.373713 18523 ts_tablet_manager.cc:1918] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:22:06.373912 18523 log.cc:1196] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/04051e10661446c49f875b50f5c94e72
I20260811 22:22:06.374378 18523 ts_tablet_manager.cc:1939] T 04051e10661446c49f875b50f5c94e72 P e1d867ffa60942929ef1b855b6b8dd75: Deleting consensus metadata
I20260811 22:22:06.375975 17537 catalog_manager.cc:4841] TS e1d867ffa60942929ef1b855b6b8dd75 (127.29.3.66:39393): tablet 04051e10661446c49f875b50f5c94e72 (table test-table [id=5c6612430b2a4d7ba584290a97c0b883]) successfully deleted
I20260811 22:22:06.378551 18528 tablet_replica.cc:330] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: stopping tablet replica
I20260811 22:22:06.378894 18528 raft_consensus.cc:2229] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:22:06.379228 18528 raft_consensus.cc:2258] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:22:06.381062 18528 ts_tablet_manager.cc:1905] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:22:06.384267 18528 ts_tablet_manager.cc:1918] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:22:06.384459 18528 log.cc:1196] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/04051e10661446c49f875b50f5c94e72
I20260811 22:22:06.384860 18528 ts_tablet_manager.cc:1939] T 04051e10661446c49f875b50f5c94e72 P b90e4fac45f548dd97f66d929ef1b4df: Deleting consensus metadata
W20260811 22:22:06.385763 17538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:34353 (error 108)
I20260811 22:22:06.386401 17538 catalog_manager.cc:4841] TS b90e4fac45f548dd97f66d929ef1b4df (127.29.3.67:34521): tablet 04051e10661446c49f875b50f5c94e72 (table test-table [id=5c6612430b2a4d7ba584290a97c0b883]) successfully deleted
I20260811 22:22:06.387131 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 17979
I20260811 22:22:06.403455 18479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:06.403052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34353 (local address 127.0.0.1:53264)
0811 22:22:06.403267 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:06.403351 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260811 22:22:06.403725 17538 catalog_manager.cc:4584] TS c38a3944b4c74fc7a63617d5d9ffb849 (127.29.3.65:34353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04051e10661446c49f875b50f5c94e72: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34353: connect: Connection refused (error 111)
I20260811 22:22:06.408650 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 18223
I20260811 22:22:06.422188 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 17504
I20260811 22:22:06.453616 29709 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 22:22:06.819324 29709 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 22:22:06.819540 29709 mini_ranger.cc:66] Stopping Ranger...
I20260811 22:22:07.165086 29709 mini_ranger.cc:68] Stopped Ranger
2026-08-11 22:22:07.165 UTC [16767] LOG: received smart shutdown request
2026-08-11 22:22:07.169 UTC [16767] LOG: background worker "logical replication launcher" (PID 16783) exited with exit code 1
2026-08-11 22:22:07.169 UTC [16771] LOG: shutting down
2026-08-11 22:22:07.346 UTC [16767] LOG: database system is shut down
2026-08-11T22:22:07Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49467 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:22:07 dist-test-slave-4tdj krb5kdc[18560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:22:07 dist-test-slave-4tdj krb5kdc[18560](info): set up 2 sockets
Aug 11 22:22:07 dist-test-slave-4tdj krb5kdc[18560](info): commencing operation
krb5kdc: starting...
W20260811 22:22:08.661264 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:22:08 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:22:08Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/rangeradmin_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/rangeradmin_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/rangerlookup_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/rangerlookup_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
I20260811 22:22:08.839226 29709 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskCdqKk8/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/postgres -l logfile start
2026-08-11 22:22:11.235 UTC [18620] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 22:22:11.235 UTC [18620] LOG: listening on IPv4 address "127.29.3.84", port 44129
2026-08-11 22:22:11.239 UTC [18620] LOG: listening on Unix socket "/tmp/.s.PGSQL.44129"
2026-08-11 22:22:11.267 UTC [18623] LOG: database system was shut down at 2026-08-11 22:22:09 UTC
2026-08-11 22:22:11.270 UTC [18620] LOG: database system is ready to accept connections
I20260811 22:22:12.364962 29709 mini_postgres.cc:96] Postgres bound to 44129
2026-08-11 22:22:12.372 UTC [18637] FATAL: database "slave" does not exist
127.29.3.84:44129 - accepting connections
I20260811 22:22:12.372922 29709 mini_ranger.cc:161] Starting Ranger...
I20260811 22:22:12.389356 29709 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 22:22:12.838850 29709 mini_ranger.cc:87] Created ranger Postgres database
I20260811 22:22:12.838936 29709 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-admin
2026-08-11 22:22:13,128 [I] DB FLAVOR :POSTGRES
2026-08-11 22:22:13,128 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:22:13,128 [I] Checking connection..
2026-08-11 22:22:13,128 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 22:22:13,402 [I] Checking connection passed.
2026-08-11 22:22:13,402 [I] --------- Verifying version history table ---------
2026-08-11 22:22:13,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:22:13,672 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:22:13,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:22:13,942 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 22:22:13,942 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 22:22:13,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 22:22:14.206 UTC [18752] WARNING: there is no transaction in progress
2026-08-11 22:22:14,218 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 22:22:14,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 22:22:14,475 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 22:22:14,475 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:22:14,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 22:22:14,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 22:22:14,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4tdj.c.gcp-upstream.internal','N') ;"
2026-08-11 22:22:15,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:22:15,485 [I] Table x_portal_user does not exist in database ranger
2026-08-11 22:22:15,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:22:15,755 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 22:22:15,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:22:15,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:22:16,262 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 22:22:16,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 22:22:17.263 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.278 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.293 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.308 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.325 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.342 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.397 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.404 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.418 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.428 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.438 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.452 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.462 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.469 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.476 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:17.483 UTC [18982] WARNING: there is no transaction in progress
2026-08-11 22:22:18,112 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 22:22:18,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 22:22:18,365 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 22:22:18,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 22:22:18,614 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 22:22:18,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 22:22:18,867 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 22:22:18,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 22:22:19,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 22:22:19,388 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4tdj.c.gcp-upstream.internal';"
2026-08-11 22:22:19,644 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 22:22:19,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 22:22:19,888 [I] DB_PATCHES have already been applied
I20260811 22:22:19.894387 29709 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-admin:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskCdqKk8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 22:22:19.894459 29709 mini_ranger.cc:193] Using host: 127.29.3.84
I20260811 22:22:19.898949 29709 mini_ranger.cc:227] Ranger admin URL: http://127.29.3.84:53705
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 10:22:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 22:22:21.153813 29709 mini_ranger.cc:160] Time spent starting Ranger: real 8.781s user 0.004s sys 0.008s
Aug 11, 2026 10:22:41 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 10:22:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 11, 2026 10:22:41 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 10:22:41 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 10:22:42 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 22:22:43.548146 29709 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.3.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.3.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/rangerkms_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.3.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/rangerkms_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.3.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.3.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.3.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/HTTP_127.29.3.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 22:22:43 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 22:22:43.640353 29709 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 22:22:43.651449 29709 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 22:22:44.524190 29709 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 22:22:44.524276 29709 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms
I20260811 22:22:44.524290 29709 mini_ranger_kms.cc:227] Using postgres at 127.29.3.84:44129
2026-08-11 22:22:44,705 [I] DB FLAVOR :POSTGRES
2026-08-11 22:22:44,705 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 22:22:44,705 [I] Checking connection
2026-08-11 22:22:44,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 22:22:44,950 [I] connection success
2026-08-11 22:22:44,950 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 22:22:44,951 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 22:22:44,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 22:22:45,213 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 22:22:45,214 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 22:22:45,214 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 22:22:45,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.3.84:44129/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 22:22:45,488 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 22:22:45.991206 29709 mini_ranger_kms.cc:310] Created kms service
I20260811 22:22:46.221935 29709 mini_ranger_kms.cc:326] Created kudu user
I20260811 22:22:46.271649 29709 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 22:22:46 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486963, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:46.701982 29709 mini_ranger_kms.cc:384] Added ranger policy
I20260811 22:22:46.702152 29709 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms:/tmp/dist-test-taskCdqKk8/build/debug/bin/postgresql.jar:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskCdqKk8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 22:22:46.702173 29709 mini_ranger_kms.cc:240] Using host: 127.29.3.84
I20260811 22:22:46.707504 29709 mini_ranger_kms.cc:276] Ranger KMS PID: 19315
I20260811 22:22:46.707592 29709 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.3.84:36823
Aug 11, 2026 10:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 10:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 10:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 10:22:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 10:22:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 10:22:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 22:22:49 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786486969, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.3.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:49.257582 29709 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.617s user 0.031s sys 0.135s
I20260811 22:22:49.257704 29709 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 22:22:49.257776 29709 mini_ranger_kms.cc:401] 127.29.3.84:36823/kms/v1/keys
Aug 11 22:22:49 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486969, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.3.84@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
Aug 11 22:22:49 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486969, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.3.84@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
Aug 11 22:22:52 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:22:52.726806 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36639
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:36639
--ranger_config_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:52.928855 19397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:52.929375 19397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:52.929477 19397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:52.935999 19397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:22:52.936255 19397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:52.936349 19397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:52.936422 19397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:22:52.936475 19397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:22:52.946069 19397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.3.84:36823/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:36639
--ranger_config_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:52.947863 19397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:52.949748 19397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:52.965250 19404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:52.965304 19403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:52.965577 19397 server_base.cc:1034] running on GCE node
W20260811 22:22:52.965895 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:52.966642 19397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:52.967993 19397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:52.969190 19397 hybrid_clock.cc:648] HybridClock initialized: now 1786486972969161 us; error 41 us; skew 500 ppm
Aug 11 22:22:52 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486972, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:52.977646 19397 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:22:52.979103 19397 webserver.cc:460] Webserver started at http://127.29.3.126:43851/ using document root <none> and password file <none>
I20260811 22:22:52.979777 19397 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:52.979843 19397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:52.980072 19397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:52 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486972, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:53.050225 19397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "0a64c109efba4950bacb6f4ba596cc37"
format_stamp: "Formatted at 2026-08-11 22:22:53 on dist-test-slave-4tdj"
server_key: "868393f2795c37e353cc4a5fd7f26c37"
server_key_iv: "c276f5e63347d63b06752e3f14966300"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:53.051258 19397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "0a64c109efba4950bacb6f4ba596cc37"
format_stamp: "Formatted at 2026-08-11 22:22:53 on dist-test-slave-4tdj"
server_key: "868393f2795c37e353cc4a5fd7f26c37"
server_key_iv: "c276f5e63347d63b06752e3f14966300"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:53.057713 19397 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260811 22:22:53.089874 19422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:53.093087 19397 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260811 22:22:53.093374 19397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "0a64c109efba4950bacb6f4ba596cc37"
format_stamp: "Formatted at 2026-08-11 22:22:53 on dist-test-slave-4tdj"
server_key: "868393f2795c37e353cc4a5fd7f26c37"
server_key_iv: "c276f5e63347d63b06752e3f14966300"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:53.093623 19397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:53.157902 19397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:53.162922 19397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:53.163244 19397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:53.180841 19397 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:36639
I20260811 22:22:53.180998 19515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:36639 every 8 connection(s)
I20260811 22:22:53.182832 19397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:22:53.183596 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 19397
I20260811 22:22:53.183797 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:22:53.184211 29709 external_mini_cluster.cc:1428] Setting key aca9b9d853761dc979e66075fdd8461d
I20260811 22:22:53.190398 19516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 11 22:22:53 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:53.204608 19516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Bootstrap starting.
I20260811 22:22:53.207530 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:53.186808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.29.3.126:36639)
0811 22:22:53.187631 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:53.187668 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:53.187763 (+ 95us) server_negotiation.cc:407] Connection header received
0811 22:22:53.188853 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:53.188887 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:53.189443 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:53.190103 (+ 660us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:53.191459 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:53.192735 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:53.194136 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:53.195037 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:53.195354 (+ 317us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:53.199516 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:53.199523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:53.199550 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:53.199609 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:53.202454 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:53.203191 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:53.203198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:53.203213 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:53.203381 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:53.203853 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:53.203857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:53.203858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:53.204552 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:53.205092 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:53.205574 (+ 482us) server_negotiation.cc:299] Negotiation successful
0811 22:22:53.206201 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":274,"threads_started":1}
I20260811 22:22:53.210381 19516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:53.213020 19516 log.cc:824] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:53.219726 19516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: No bootstrap required, opened a new log
I20260811 22:22:53.228144 19516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:53.228884 19516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:53.229038 19516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a64c109efba4950bacb6f4ba596cc37, State: Initialized, Role: FOLLOWER
I20260811 22:22:53.229965 19516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:53.230337 19516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:22:53.230476 19516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:22:53.230671 19516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:53.232913 19516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:53.233641 19516 leader_election.cc:302] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a64c109efba4950bacb6f4ba596cc37; no voters:
I20260811 22:22:53.234380 19516 leader_election.cc:288] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:22:53.234561 19522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:22:53.234936 19522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 LEADER]: Becoming Leader. State: Replica: 0a64c109efba4950bacb6f4ba596cc37, State: Running, Role: LEADER
I20260811 22:22:53.235597 19522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:53.237008 19516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:22:53.238245 19523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a64c109efba4950bacb6f4ba596cc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } } }
I20260811 22:22:53.238209 19525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a64c109efba4950bacb6f4ba596cc37. Latest consensus state: current_term: 1 leader_uuid: "0a64c109efba4950bacb6f4ba596cc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } } }
I20260811 22:22:53.238477 19525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: This master's current role is: LEADER
I20260811 22:22:53.238477 19523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: This master's current role is: LEADER
I20260811 22:22:53.241248 19516 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 22:22:53.242225 19535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:22:53.254686 19535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:22:53.266398 19535 catalog_manager.cc:1347] Generated new cluster ID: f2ebc828c0834aa0b13198a3a4d81670
I20260811 22:22:53.266641 19535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:22:53.305955 19535 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:22:53.308396 19535 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:22:53.326664 19535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Generated new TSK 0
I20260811 22:22:53.328250 19535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:22:54.367056 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36639
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:54.490829 19584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:54.491180 19584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:54.491248 19584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:54.495707 19584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:54.495776 19584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:54.495859 19584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:22:54.501741 19584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.3.84:36823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:54.503032 19584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:54.504148 19584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:54.517333 19595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:54.517342 19594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:54.517597 19597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:54.517589 19584 server_base.cc:1034] running on GCE node
I20260811 22:22:54.518119 19584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:54.518846 19584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:54.520023 19584 hybrid_clock.cc:648] HybridClock initialized: now 1786486974519989 us; error 47 us; skew 500 ppm
Aug 11 22:22:54 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:54.522837 19584 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:22:54.523844 19584 webserver.cc:460] Webserver started at http://127.29.3.65:43081/ using document root <none> and password file <none>
I20260811 22:22:54.524613 19584 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:54.524741 19584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:54.525038 19584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:54 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:54.542065 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "badbb18c828843b69721aead91bdd343"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "144bad49c78c9f357e438c04326a4e14"
server_key_iv: "8faa30363382c9528abf60704d252efb"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:54.543020 19584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "badbb18c828843b69721aead91bdd343"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "144bad49c78c9f357e438c04326a4e14"
server_key_iv: "8faa30363382c9528abf60704d252efb"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:54.552060 19584 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260811 22:22:54.573038 19611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:54.575027 19584 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260811 22:22:54.575281 19584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "badbb18c828843b69721aead91bdd343"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "144bad49c78c9f357e438c04326a4e14"
server_key_iv: "8faa30363382c9528abf60704d252efb"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:54.575551 19584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:54.600482 19584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:54.604637 19584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:54.604918 19584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:54.605818 19584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:54.607069 19584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:54.607219 19584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:54.607358 19584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:54.607446 19584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:54.618145 19584 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:45599
I20260811 22:22:54.619457 19584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:22:54.626176 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 19584
I20260811 22:22:54.626353 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:22:54.626686 29709 external_mini_cluster.cc:1428] Setting key 3e618763eda6b51f5469a62e1840643e
I20260811 22:22:54.635778 19812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:45599 every 8 connection(s)
Aug 11 22:22:54 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:54.654515 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:54.638090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:42335 (local address 127.29.3.126:36639)
0811 22:22:54.638474 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:54.638479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:54.639484 (+ 1005us) server_negotiation.cc:407] Connection header received
0811 22:22:54.641434 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:54.641438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:54.641514 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:54.641587 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:54.643500 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:54.644315 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:54.646121 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:54.646519 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:54.646606 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:54.649906 (+ 3300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:54.649910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:54.649914 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:54.649949 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:54.651860 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:54.652533 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:54.652537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:54.652539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:54.652632 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:54.653045 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:54.653049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:54.653051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:54.653300 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:54.653375 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:54.654132 (+ 757us) server_negotiation.cc:299] Negotiation successful
0811 22:22:54.654322 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":135,"threads_started":1}
I20260811 22:22:54.656567 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:54.638739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36639 (local address 127.29.3.65:42335)
0811 22:22:54.639250 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:54.639307 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:54.641117 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:54.642088 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:54.642098 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:54.642530 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:54.643303 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:54.643327 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:54.644500 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:54.644503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:54.645974 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:54.645986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:54.646844 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:54.646849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:54.647078 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:54.647783 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:54.647804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:54.649667 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:54.652001 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:54.652008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:54.652029 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:54.652411 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:54.652738 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:54.652740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:54.652742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:54.652937 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:54.653460 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:54.653467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:54.653839 (+ 372us) client_negotiation.cc:769] Sending connection context
0811 22:22:54.654994 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0811 22:22:54.655454 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260811 22:22:54.658283 19813 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36639
I20260811 22:22:54.658574 19813 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:54.659190 19813 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:54.661167 19450 ts_manager.cc:194] Registered new tserver with Master: badbb18c828843b69721aead91bdd343 (127.29.3.65:45599)
I20260811 22:22:54.662798 19450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:42335
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:22:54.698238 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36639
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:22:54 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 22:22:54.875337 19828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:54.875854 19828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:54.875976 19828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:54.882414 19828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:54.882579 19828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:54.882769 19828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:22:54.891073 19828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.3.84:36823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:54.892840 19828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:54.894337 19828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:54.907868 19838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:54.908828 19837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:54.909134 19828 server_base.cc:1034] running on GCE node
W20260811 22:22:54.908929 19842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:54.909946 19828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:54.910915 19828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:54.912078 19828 hybrid_clock.cc:648] HybridClock initialized: now 1786486974912011 us; error 85 us; skew 500 ppm
Aug 11 22:22:54 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:54.915649 19828 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:22:54.916843 19828 webserver.cc:460] Webserver started at http://127.29.3.66:41291/ using document root <none> and password file <none>
I20260811 22:22:54.917645 19828 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:54.917795 19828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:54.918131 19828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:54 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:54.936023 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "4e5a08aca64f44f486f34044a96fe38e"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "78cff251fba1a404ad156ace06aac26c"
server_key_iv: "4846141b01d489599b30bee6416344dd"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:54.936976 19828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "4e5a08aca64f44f486f34044a96fe38e"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "78cff251fba1a404ad156ace06aac26c"
server_key_iv: "4846141b01d489599b30bee6416344dd"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:54.948230 19828 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.000s sys 0.004s
I20260811 22:22:54.977347 19858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:54.979395 19828 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260811 22:22:54.979575 19828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "4e5a08aca64f44f486f34044a96fe38e"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "78cff251fba1a404ad156ace06aac26c"
server_key_iv: "4846141b01d489599b30bee6416344dd"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:54.979763 19828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:55.030220 19828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:55.034215 19828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:55.034520 19828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:55.035357 19828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:55.036497 19828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:55.036612 19828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:55.036721 19828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:55.036790 19828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:55.048509 19828 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:38447
I20260811 22:22:55.049806 19828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:22:55.052388 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:38447 every 8 connection(s)
I20260811 22:22:55.056104 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 19828
I20260811 22:22:55.056241 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:22:55.056537 29709 external_mini_cluster.cc:1428] Setting key 52e5d87bd18b8e2e873f40e42c80e846
Aug 11 22:22:55 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:55.069595 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:55.054391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:39127 (local address 127.29.3.126:36639)
0811 22:22:55.054532 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:55.054536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:55.055590 (+ 1054us) server_negotiation.cc:407] Connection header received
0811 22:22:55.056795 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:55.056800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:55.056862 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:55.056969 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:55.058605 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.059413 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.061270 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.061633 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.061725 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:55.065114 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:55.065120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:55.065122 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:55.065173 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:55.066992 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:55.067714 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:55.067719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:55.067720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:55.067817 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:55.068242 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:55.068245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:55.068246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:55.068474 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:55.068551 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:55.069287 (+ 736us) server_negotiation.cc:299] Negotiation successful
0811 22:22:55.069429 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:22:55.071794 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:55.054861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36639 (local address 127.29.3.66:39127)
0811 22:22:55.055387 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:55.055443 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:55.056514 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:55.057202 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:55.057212 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:55.057616 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:55.058413 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:55.058438 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.059598 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.059601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:55.061095 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:55.061110 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.061974 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.061979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:55.062222 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:55.062999 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:55.063023 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:55.064866 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:55.067143 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:55.067169 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:55.067190 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:55.067581 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:55.067926 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:55.067929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:55.067931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:55.068123 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:55.068619 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:55.068626 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:55.068991 (+ 365us) client_negotiation.cc:769] Sending connection context
0811 22:22:55.070146 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0811 22:22:55.070595 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":140,"threads_started":1}
I20260811 22:22:55.073766 20047 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36639
I20260811 22:22:55.074057 20047 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:55.074900 20047 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:22:55.076311 19450 ts_manager.cc:194] Registered new tserver with Master: 4e5a08aca64f44f486f34044a96fe38e (127.29.3.66:38447)
I20260811 22:22:55.076972 19450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:39127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:22:55.106566 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36639
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:55.284912 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:55.285902 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:55.286043 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:55.292366 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:55.292522 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:55.292662 20053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:22:55.300920 20053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.3.84:36823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:55.302618 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:55.304312 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:55.319235 20066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:55.319463 20062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:55.319741 20061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:55.319845 20053 server_base.cc:1034] running on GCE node
I20260811 22:22:55.321089 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:55.321959 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:55.330365 20053 hybrid_clock.cc:648] HybridClock initialized: now 1786486975327705 us; error 2651 us; skew 500 ppm
Aug 11 22:22:55 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:55.334415 20053 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:22:55.335657 20053 webserver.cc:460] Webserver started at http://127.29.3.67:38227/ using document root <none> and password file <none>
I20260811 22:22:55.336404 20053 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:55.336498 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:22:55.336769 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 22:22:55 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:55.353963 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "73946b78ced4491fb7759cf18360aa13"
format_stamp: "Formatted at 2026-08-11 22:22:55 on dist-test-slave-4tdj"
server_key: "1666350b94f402362338e7aba60939f0"
server_key_iv: "9ed390cd61cce317befaf0f0a6d435ce"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:55.354938 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "73946b78ced4491fb7759cf18360aa13"
format_stamp: "Formatted at 2026-08-11 22:22:55 on dist-test-slave-4tdj"
server_key: "1666350b94f402362338e7aba60939f0"
server_key_iv: "9ed390cd61cce317befaf0f0a6d435ce"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:55.362159 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
Aug 11 22:22:55 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486974, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:55.394759 20083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:55.396785 20053 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 22:22:55.396931 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "73946b78ced4491fb7759cf18360aa13"
format_stamp: "Formatted at 2026-08-11 22:22:55 on dist-test-slave-4tdj"
server_key: "1666350b94f402362338e7aba60939f0"
server_key_iv: "9ed390cd61cce317befaf0f0a6d435ce"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:55.397069 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:55.444684 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:55.449229 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:55.449486 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:55.450532 20053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:55.451748 20053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:55.451864 20053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:55.451956 20053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:55.452013 20053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:55.467164 20053 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:42811
I20260811 22:22:55.468529 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:22:55.469219 20270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:42811 every 8 connection(s)
I20260811 22:22:55.475183 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 20053
I20260811 22:22:55.475310 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:22:55.475589 29709 external_mini_cluster.cc:1428] Setting key 3c4c1f21bede281c0912cd818c2313da
Aug 11 22:22:55 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:55.492981 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:55.473462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:42489 (local address 127.29.3.126:36639)
0811 22:22:55.473588 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:55.473593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:55.474115 (+ 522us) server_negotiation.cc:407] Connection header received
0811 22:22:55.478862 (+ 4747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:55.478866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:55.478925 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:55.478972 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:55.480703 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.481563 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.483474 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.483813 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.483887 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:55.487027 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:55.487031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:55.487034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:55.487064 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:55.490440 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:55.491111 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:55.491114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:55.491116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:55.491214 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:55.491613 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:55.491616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:55.491618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:55.491856 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:55.491932 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:55.492682 (+ 750us) server_negotiation.cc:299] Negotiation successful
0811 22:22:55.492816 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:22:55.495117 20276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:55.472283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36639 (local address 127.29.3.67:42489)
0811 22:22:55.473950 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:55.474010 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:55.478576 (+ 4566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:55.479427 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:55.479437 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:55.479829 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:55.480525 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:55.480543 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.481824 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.481827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:55.483339 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:55.483352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:55.484105 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:55.484109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:55.484325 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:55.484984 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:55.485006 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:55.486804 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:55.490574 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:55.490582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:55.490604 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:55.490996 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:55.491316 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:55.491319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:55.491321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:55.491516 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:55.491999 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:55.492007 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:55.492386 (+ 379us) client_negotiation.cc:769] Sending connection context
0811 22:22:55.493445 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0811 22:22:55.493859 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"thread_start_us":166,"threads_started":1}
I20260811 22:22:55.496919 20271 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36639
I20260811 22:22:55.497212 20271 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:55.497862 20271 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:55.499212 19450 ts_manager.cc:194] Registered new tserver with Master: 73946b78ced4491fb7759cf18360aa13 (127.29.3.67:42811)
I20260811 22:22:55.499998 19450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:42489
I20260811 22:22:55.501374 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:22:55.502134 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 19584
I20260811 22:22:55.546586 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 19828
I20260811 22:22:55.558121 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 20053
I20260811 22:22:55.593698 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 19397
I20260811 22:22:55.618166 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36639
--webserver_interface=127.29.3.126
--webserver_port=43851
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:36639
--ranger_config_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:22:55.732780 20278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:55.733063 20278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:55.733103 20278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:55.737093 20278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:22:55.737159 20278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:55.737176 20278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:55.737191 20278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:22:55.737205 20278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:22:55.742781 20278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:36639
--ranger_config_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=43851
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:55.743861 20278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:55.744853 20278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:55.753659 20288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:55.753806 20278 server_base.cc:1034] running on GCE node
W20260811 22:22:55.753684 20289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:55.753690 20295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:55.754504 20278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:55.755635 20278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:55.756801 20278 hybrid_clock.cc:648] HybridClock initialized: now 1786486975756783 us; error 35 us; skew 500 ppm
Aug 11 22:22:55 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:55.759603 20278 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:22:55.760571 20278 webserver.cc:460] Webserver started at http://127.29.3.126:43851/ using document root <none> and password file <none>
I20260811 22:22:55.761116 20278 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:55.761196 20278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:22:56 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486975, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:56.845294 20278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:22:56.848256 20309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:56.849565 20278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:22:56.849746 20278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "0a64c109efba4950bacb6f4ba596cc37"
format_stamp: "Formatted at 2026-08-11 22:22:53 on dist-test-slave-4tdj"
server_key: "868393f2795c37e353cc4a5fd7f26c37"
server_key_iv: "c276f5e63347d63b06752e3f14966300"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:56.850347 20278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:56.884187 20278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:56.887789 20278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:56.887975 20278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:56.895388 20278 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:36639
I20260811 22:22:56.895406 20395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:36639 every 8 connection(s)
I20260811 22:22:56.896526 20278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:22:56.901031 20396 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:22:56.902783 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Bootstrap starting.
I20260811 22:22:56.906680 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 20278
I20260811 22:22:56.907181 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:45599
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:36639
--webserver_port=43081
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:22:56.919909 20396 log.cc:824] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:56.927095 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:22:56.927498 20396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Bootstrap complete.
I20260811 22:22:56.931582 20396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:56.932004 20396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a64c109efba4950bacb6f4ba596cc37, State: Initialized, Role: FOLLOWER
I20260811 22:22:56.932533 20396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:56.932657 20396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:22:56.932721 20396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:22:56.932816 20396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:22:56.934851 20396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:56.935191 20396 leader_election.cc:302] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a64c109efba4950bacb6f4ba596cc37; no voters:
I20260811 22:22:56.935499 20396 leader_election.cc:288] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:22:56.935642 20406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:22:56.935820 20406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [term 2 LEADER]: Becoming Leader. State: Replica: 0a64c109efba4950bacb6f4ba596cc37, State: Running, Role: LEADER
I20260811 22:22:56.936193 20406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } }
I20260811 22:22:56.936805 20396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:22:56.937527 20410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a64c109efba4950bacb6f4ba596cc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } } }
I20260811 22:22:56.937669 20410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: This master's current role is: LEADER
I20260811 22:22:56.938406 20411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a64c109efba4950bacb6f4ba596cc37. Latest consensus state: current_term: 2 leader_uuid: "0a64c109efba4950bacb6f4ba596cc37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a64c109efba4950bacb6f4ba596cc37" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36639 } } }
I20260811 22:22:56.938495 20411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37 [sys.catalog]: This master's current role is: LEADER
I20260811 22:22:56.938843 20420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:22:56.939206 20396 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 22:22:56.945678 20420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:22:56.946727 20420 catalog_manager.cc:1259] Loaded cluster ID: f2ebc828c0834aa0b13198a3a4d81670
I20260811 22:22:56.946833 20420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:22:56.949437 20420 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:22:56.950234 20420 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Loaded TSK: 0
I20260811 22:22:56.951617 20420 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:22:57.056748 20398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:57.057258 20398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:57.057401 20398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:57.063053 20398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:57.063143 20398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:57.063237 20398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:22:57.071242 20398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:45599
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=43081
--tserver_master_addrs=127.29.3.126:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:57.072753 20398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:57.074167 20398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:57.088177 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:57.088634 20468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:57.088850 20398 server_base.cc:1034] running on GCE node
W20260811 22:22:57.088864 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:57.089382 20398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:57.090230 20398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:57.091477 20398 hybrid_clock.cc:648] HybridClock initialized: now 1786486977091408 us; error 95 us; skew 500 ppm
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:57.095209 20398 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:22:57.096258 20398 webserver.cc:460] Webserver started at http://127.29.3.65:43081/ using document root <none> and password file <none>
I20260811 22:22:57.096856 20398 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:57.096948 20398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:57.119668 20398 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:22:57.123131 20481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.124739 20398 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:22:57.124903 20398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "badbb18c828843b69721aead91bdd343"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "144bad49c78c9f357e438c04326a4e14"
server_key_iv: "8faa30363382c9528abf60704d252efb"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:57.125667 20398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:57.149322 20398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:57.153299 20398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:57.153529 20398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:57.154332 20398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:57.155426 20398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:57.155488 20398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.155612 20398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:57.155649 20398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.166790 20398 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:45599
I20260811 22:22:57.166831 20670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:45599 every 8 connection(s)
I20260811 22:22:57.168191 20398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:22:57.176635 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 20398
I20260811 22:22:57.177156 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:38447
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:36639
--webserver_port=41291
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:57.201926 20674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.176781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36639 (local address 127.29.3.65:42155)
0811 22:22:57.182184 (+ 5403us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:57.182252 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:57.183557 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:57.185786 (+ 2229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:57.185796 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:57.186178 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:57.186914 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:57.186937 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.188914 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.188918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:57.190380 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:57.190394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.191752 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.191758 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:57.191971 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.192690 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:57.192729 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:57.195129 (+ 2400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:57.197848 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:57.197854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:57.197869 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:57.198219 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:57.198693 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:57.198696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:57.198698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:57.198908 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:57.199906 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:57.199914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:57.200252 (+ 338us) client_negotiation.cc:769] Sending connection context
0811 22:22:57.200464 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 22:22:57.200842 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5136,"thread_start_us":170,"threads_started":1}
I20260811 22:22:57.203786 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.179431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:42155 (local address 127.29.3.126:36639)
0811 22:22:57.181821 (+ 2390us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:57.181848 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:57.182416 (+ 568us) server_negotiation.cc:407] Connection header received
0811 22:22:57.184164 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:57.184184 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:57.184700 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:57.185109 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:57.187141 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.188744 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.190519 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.191295 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.191481 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.195405 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:57.195413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:57.195434 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:57.195473 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:57.197680 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.198397 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.198404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.198410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.198572 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.199032 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.199036 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.199038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.199584 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:57.199739 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:57.202515 (+ 2776us) server_negotiation.cc:299] Negotiation successful
0811 22:22:57.202844 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2202,"thread_start_us":128,"threads_started":1}
I20260811 22:22:57.207500 20672 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36639
I20260811 22:22:57.207782 20672 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.208453 20672 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
W20260811 22:22:57.212401 20672 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:36639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 22:22:57.212546 20672 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.212769 20672 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:57.213887 20672 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.214084 20672 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
W20260811 22:22:57.214995 20672 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:22:57.347368 20675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:57.347817 20675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:57.347898 20675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:57.353960 20675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:57.354049 20675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:57.354140 20675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:22:57.360587 20675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:38447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=41291
--tserver_master_addrs=127.29.3.126:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:57.361876 20675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:57.363097 20675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:57.375006 20685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:57.375640 20675 server_base.cc:1034] running on GCE node
W20260811 22:22:57.376045 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:57.376313 20684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:57.376881 20675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:57.377764 20675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:57.378981 20675 hybrid_clock.cc:648] HybridClock initialized: now 1786486977378894 us; error 100 us; skew 500 ppm
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:57.382970 20675 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:22:57.384282 20675 webserver.cc:460] Webserver started at http://127.29.3.66:41291/ using document root <none> and password file <none>
I20260811 22:22:57.385131 20675 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:57.385299 20675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:57.419799 20675 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260811 22:22:57.423386 20706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.424945 20675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260811 22:22:57.425081 20675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "4e5a08aca64f44f486f34044a96fe38e"
format_stamp: "Formatted at 2026-08-11 22:22:54 on dist-test-slave-4tdj"
server_key: "78cff251fba1a404ad156ace06aac26c"
server_key_iv: "4846141b01d489599b30bee6416344dd"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:57.425683 20675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:57.441867 20675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:57.445653 20675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:57.445935 20675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:57.446828 20675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:57.447788 20675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:57.447841 20675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.447896 20675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:57.447925 20675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.458165 20675 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:38447
I20260811 22:22:57.458197 20901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:38447 every 8 connection(s)
I20260811 22:22:57.459838 20675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:22:57.469020 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 20675
I20260811 22:22:57.469502 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.3.84:36823/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:42811
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:36639
--webserver_port=38227
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:57.477576 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.465056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:41383 (local address 127.29.3.126:36639)
0811 22:22:57.465190 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:57.465195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:57.465213 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:22:57.465807 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:57.465811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:57.465869 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:57.465924 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:57.467676 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.468494 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.470095 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.470457 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.470534 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.473366 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:57.473370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:57.473372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:57.473406 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:57.475007 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.475649 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.475653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.475654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.475767 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.476403 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.476406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.476408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.476604 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:57.476695 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:57.477256 (+ 561us) server_negotiation.cc:299] Negotiation successful
0811 22:22:57.477394 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:22:57.478493 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.463961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36639 (local address 127.29.3.66:41383)
0811 22:22:57.464461 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:57.464515 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:57.465538 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:57.466393 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:57.466402 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:57.466766 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:57.467478 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:57.467501 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.468668 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.468671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:57.469949 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:57.469961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.470740 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.470743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:57.470936 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.471570 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:57.471595 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:57.473117 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:57.475128 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:57.475135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:57.475154 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:57.475521 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:57.476132 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:57.476135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:57.476137 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:57.476303 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:57.476721 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:57.476728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:57.477003 (+ 275us) client_negotiation.cc:769] Sending connection context
0811 22:22:57.477183 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 22:22:57.477516 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":112,"threads_started":1}
I20260811 22:22:57.479647 20903 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36639
I20260811 22:22:57.479900 20903 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.480479 20903 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
W20260811 22:22:57.482216 20903 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:36639 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 22:22:57.482367 20903 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.482545 20903 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:57.483493 20903 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.483685 20903 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
W20260811 22:22:57.484771 20903 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:22:57.618477 20908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:22:57.618882 20908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:22:57.619037 20908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:22:57.625082 20908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:22:57.625254 20908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:22:57.625394 20908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:22:57.633430 20908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.3.84:36823/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:42811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=38227
--tserver_master_addrs=127.29.3.126:36639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:22:57.634956 20908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:22:57.636389 20908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:22:57.649269 20917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:57.649283 20918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:22:57.649832 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:22:57.650117 20908 server_base.cc:1034] running on GCE node
I20260811 22:22:57.650589 20908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:22:57.651306 20908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:22:57.652501 20908 hybrid_clock.cc:648] HybridClock initialized: now 1786486977652465 us; error 52 us; skew 500 ppm
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:57.656430 20908 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:22:57.657683 20908 webserver.cc:460] Webserver started at http://127.29.3.67:38227/ using document root <none> and password file <none>
I20260811 22:22:57.658499 20908 fs_manager.cc:359] Metadata directory not provided
I20260811 22:22:57.658665 20908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:57.677564 20908 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:22:57.682015 20935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.684013 20908 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260811 22:22:57.684175 20908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "73946b78ced4491fb7759cf18360aa13"
format_stamp: "Formatted at 2026-08-11 22:22:55 on dist-test-slave-4tdj"
server_key: "1666350b94f402362338e7aba60939f0"
server_key_iv: "9ed390cd61cce317befaf0f0a6d435ce"
server_key_version: "kuduclusterkey@0"
I20260811 22:22:57.684960 20908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:22:57.733258 20908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:22:57.736935 20908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:22:57.737166 20908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:22:57.738050 20908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:22:57.739231 20908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:22:57.739297 20908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.739355 20908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:22:57.739385 20908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:22:57.750926 20908 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:42811
I20260811 22:22:57.751107 21116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:42811 every 8 connection(s)
I20260811 22:22:57.752434 20908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:22:57.756875 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 20908
Aug 11 22:22:57 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486977, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:22:57.771417 20677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.754466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:44575 (local address 127.29.3.126:36639)
0811 22:22:57.754611 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:57.754616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:57.759001 (+ 4385us) server_negotiation.cc:407] Connection header received
0811 22:22:57.760240 (+ 1239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:57.760243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:57.760294 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:57.760382 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:57.761865 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.762459 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.763982 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.764267 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.764372 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.767177 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:57.767182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:57.767184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:57.767236 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:57.768811 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.769442 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.769446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.769448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.769556 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.769992 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.769996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.769998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.770206 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:57.770282 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:57.771106 (+ 824us) server_negotiation.cc:299] Negotiation successful
0811 22:22:57.771246 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:22:57.772663 21130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.755077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36639 (local address 127.29.3.67:44575)
0811 22:22:57.758814 (+ 3737us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:57.758879 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:57.759969 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:57.760614 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:57.760622 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:57.760998 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:22:57.761720 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:57.761738 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.762608 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.762612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:57.763863 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:57.763873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.764436 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.764440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:57.764645 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.765205 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:22:57.765236 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:22:57.766971 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:22:57.768936 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:57.768943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:57.768961 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:57.769328 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:57.769665 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:22:57.769669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:22:57.769670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:22:57.769871 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:22:57.770408 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:22:57.770416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:22:57.770805 (+ 389us) client_negotiation.cc:769] Sending connection context
0811 22:22:57.771055 (+ 250us) client_negotiation.cc:240] Negotiation successful
0811 22:22:57.771448 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3475,"thread_start_us":164,"threads_started":1}
I20260811 22:22:57.773965 21117 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36639
I20260811 22:22:57.774250 21117 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:57.774895 21117 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:57.775046 21138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:57.757581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.29.3.126:36639)
0811 22:22:57.760453 (+ 2872us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:57.760461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:57.760476 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:22:57.760518 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:57.760521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:57.760572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:57.760667 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:57.761558 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.762410 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.763625 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:57.763993 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:57.764097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:57.765657 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:57.765661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:57.765664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:57.765691 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:57.767866 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.768605 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.768608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.768610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.768709 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:57.769836 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:57.769840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:57.769842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:57.770086 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:57.770323 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:57.774737 (+ 4414us) server_negotiation.cc:299] Negotiation successful
0811 22:22:57.774871 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2790,"thread_start_us":97,"threads_started":1}
I20260811 22:22:57.777032 20340 ts_manager.cc:194] Registered new tserver with Master: 73946b78ced4491fb7759cf18360aa13 (127.29.3.67:42811)
I20260811 22:22:57.778184 20340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:44575
Aug 11 22:22:58 dist-test-slave-4tdj krb5kdc[18560](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786486978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:22:58.215253 20672 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:58.215600 20672 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:58.216557 20340 ts_manager.cc:194] Registered new tserver with Master: badbb18c828843b69721aead91bdd343 (127.29.3.65:45599)
I20260811 22:22:58.217377 20340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:42155
I20260811 22:22:58.485008 20903 heartbeater.cc:464] Registering TS with master...
I20260811 22:22:58.485337 20903 heartbeater.cc:511] Master 127.29.3.126:36639 requested a full tablet report, sending...
I20260811 22:22:58.486387 20340 ts_manager.cc:194] Registered new tserver with Master: 4e5a08aca64f44f486f34044a96fe38e (127.29.3.66:38447)
I20260811 22:22:58.486989 20340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:41383
Aug 11 22:22:58 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:58.497717 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:22:58.525750 21154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.499742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.29.3.126:36639)
0811 22:22:58.500358 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.500363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.500375 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:22:58.500414 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.500417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.500462 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.500537 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:22:58.501457 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.502288 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.513708 (+ 11420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.514166 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.514320 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.520232 (+ 5912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:22:58.520238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:22:58.520241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:22:58.520286 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:22:58.522212 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:58.523311 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:58.523316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:58.523318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:58.523440 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:22:58.524154 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:22:58.524158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:22:58.524160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:22:58.524436 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:22:58.524578 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.525387 (+ 809us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.525541 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":99,"threads_started":1}
I20260811 22:22:58.531123 20340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:22:58.535980 20340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:22:58.561895 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.550472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:45599 (local address 127.0.0.1:39668)
0811 22:22:58.553597 (+ 3125us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.553661 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:58.554076 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:58.556013 (+ 1937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:58.556020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:58.556045 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:58.556202 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.556209 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.557601 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.557604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.559896 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.559909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.561269 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.561272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.561398 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.561441 (+ 43us) client_negotiation.cc:769] Sending connection context
0811 22:22:58.561550 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 22:22:58.561682 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3012,"thread_start_us":126,"threads_started":1}
I20260811 22:22:58.563660 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.551706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.29.3.65:45599)
0811 22:22:58.553377 (+ 1671us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.553385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.553773 (+ 388us) server_negotiation.cc:407] Connection header received
0811 22:22:58.554551 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.554558 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.555678 (+ 1120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.555833 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:58.556332 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.557476 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.560042 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.560932 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.561033 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.561204 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.563389 (+ 2185us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.563514 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":62,"threads_started":1}
I20260811 22:22:58.567282 20530 tablet_service.cc:1467] Processing CreateTablet for tablet c0352621bc894b4d9d8842c2936dbd61 (DEFAULT_TABLE table=test-table [id=37a01ce148844581b0ffe46ada86ccb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:22:58.569482 20530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0352621bc894b4d9d8842c2936dbd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:58.573167 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.550253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:42811 (local address 127.0.0.1:60054)
0811 22:22:58.553817 (+ 3564us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.553828 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:58.553948 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:58.558236 (+ 4288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:58.558240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:58.558258 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:58.558369 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.558376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.569966 (+ 11590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.569970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.571088 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.571102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.572726 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.572730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.572825 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.572853 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 22:22:58.572920 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 22:22:58.573007 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2752,"thread_start_us":97,"threads_started":1}
I20260811 22:22:58.576241 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.550382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.29.3.67:42811)
0811 22:22:58.557599 (+ 7217us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.557607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.557632 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:22:58.557709 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.557714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.557928 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.558079 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:58.567050 (+ 8971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.568135 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.571267 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.572361 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.572473 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.572653 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.575949 (+ 3296us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.576086 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7084,"thread_start_us":77,"threads_started":1}
I20260811 22:22:58.577919 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.551306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38447 (local address 127.0.0.1:39194)
0811 22:22:58.557235 (+ 5929us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.557249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:58.557336 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:58.568289 (+ 10953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:58.568292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:58.568310 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:58.568411 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.568418 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.573747 (+ 5329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.573750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.574787 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.574799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.577477 (+ 2678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.577481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.577584 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.577629 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 22:22:58.577685 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 22:22:58.577771 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5861,"thread_start_us":95,"threads_started":1}
I20260811 22:22:58.578948 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.551484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.29.3.66:38447)
0811 22:22:58.566526 (+ 15042us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.566533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.566560 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:22:58.566654 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.566659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.566869 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.568599 (+ 1730us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:58.568778 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.569814 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.574956 (+ 5142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.575788 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.578448 (+ 2660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.578640 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.578699 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.578806 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14915,"thread_start_us":94,"threads_started":1}
I20260811 22:22:58.579852 21009 tablet_service.cc:1467] Processing CreateTablet for tablet c0352621bc894b4d9d8842c2936dbd61 (DEFAULT_TABLE table=test-table [id=37a01ce148844581b0ffe46ada86ccb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:22:58.583940 20760 tablet_service.cc:1467] Processing CreateTablet for tablet c0352621bc894b4d9d8842c2936dbd61 (DEFAULT_TABLE table=test-table [id=37a01ce148844581b0ffe46ada86ccb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:22:58.585186 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0352621bc894b4d9d8842c2936dbd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:58.587100 21009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0352621bc894b4d9d8842c2936dbd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:22:58.611706 21174 tablet_bootstrap.cc:492] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Bootstrap starting.
I20260811 22:22:58.615824 21177 tablet_bootstrap.cc:492] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: Bootstrap starting.
I20260811 22:22:58.616308 21179 tablet_bootstrap.cc:492] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Bootstrap starting.
I20260811 22:22:58.618947 21177 tablet_bootstrap.cc:654] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:58.621366 21179 tablet_bootstrap.cc:654] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:58.621526 21174 tablet_bootstrap.cc:654] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Neither blocks nor log segments found. Creating new log.
I20260811 22:22:58.622623 21174 log.cc:824] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:58.623576 21177 log.cc:824] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:58.624416 21179 log.cc:824] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Log is configured to *not* fsync() on all Append() calls
I20260811 22:22:58.626422 21179 tablet_bootstrap.cc:492] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: No bootstrap required, opened a new log
I20260811 22:22:58.626657 21179 ts_tablet_manager.cc:1397] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260811 22:22:58.626834 21174 tablet_bootstrap.cc:492] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: No bootstrap required, opened a new log
I20260811 22:22:58.627022 21174 ts_tablet_manager.cc:1397] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20260811 22:22:58.627032 21177 tablet_bootstrap.cc:492] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: No bootstrap required, opened a new log
I20260811 22:22:58.627260 21177 ts_tablet_manager.cc:1397] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260811 22:22:58.631049 21174 raft_consensus.cc:348] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.631417 21174 raft_consensus.cc:374] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:58.631490 21174 raft_consensus.cc:729] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e5a08aca64f44f486f34044a96fe38e, State: Initialized, Role: FOLLOWER
I20260811 22:22:58.631671 21177 raft_consensus.cc:348] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.631963 21177 raft_consensus.cc:374] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:58.632022 21177 raft_consensus.cc:729] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: badbb18c828843b69721aead91bdd343, State: Initialized, Role: FOLLOWER
I20260811 22:22:58.632175 21174 consensus_queue.cc:260] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.632526 21177 consensus_queue.cc:260] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.632696 21179 raft_consensus.cc:348] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.632980 21179 raft_consensus.cc:374] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:22:58.633036 21179 raft_consensus.cc:729] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73946b78ced4491fb7759cf18360aa13, State: Initialized, Role: FOLLOWER
I20260811 22:22:58.633538 21174 ts_tablet_manager.cc:1428] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260811 22:22:58.633538 21177 ts_tablet_manager.cc:1428] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 22:22:58.633491 21179 consensus_queue.cc:260] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.633934 20672 heartbeater.cc:503] Master 127.29.3.126:36639 was elected leader, sending a full tablet report...
I20260811 22:22:58.634457 21179 ts_tablet_manager.cc:1428] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260811 22:22:58.634760 21117 heartbeater.cc:503] Master 127.29.3.126:36639 was elected leader, sending a full tablet report...
I20260811 22:22:58.635386 20903 heartbeater.cc:503] Master 127.29.3.126:36639 was elected leader, sending a full tablet report...
W20260811 22:22:58.677155 20673 tablet.cc:2369] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:22:58.708997 21189 raft_consensus.cc:482] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:22:58.709259 21189 raft_consensus.cc:504] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.710704 21189 leader_election.cc:288] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers badbb18c828843b69721aead91bdd343 (127.29.3.65:45599), 4e5a08aca64f44f486f34044a96fe38e (127.29.3.66:38447)
W20260811 22:22:58.713620 20904 tablet.cc:2369] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:22:58.716297 21167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.711437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:42597 (local address 127.29.3.66:38447)
0811 22:22:58.711561 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.711566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.711881 (+ 315us) server_negotiation.cc:407] Connection header received
0811 22:22:58.712058 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.712062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.712123 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.712169 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:58.712598 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.713321 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.714917 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.715534 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.715617 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.715685 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.716052 (+ 367us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.716158 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:22:58.716739 21195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.711026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38447 (local address 127.29.3.67:42597)
0811 22:22:58.711795 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.711808 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:58.711952 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:58.712303 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:58.712305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:58.712321 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:58.712472 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.712477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.713465 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.713467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.714777 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.714789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.715782 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.715785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.715892 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.715919 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 22:22:58.716502 (+ 583us) client_negotiation.cc:240] Negotiation successful
0811 22:22:58.716580 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260811 22:22:58.717404 20835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0352621bc894b4d9d8842c2936dbd61" candidate_uuid: "73946b78ced4491fb7759cf18360aa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5a08aca64f44f486f34044a96fe38e" is_pre_election: true
I20260811 22:22:58.717530 21171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.711482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:51609 (local address 127.29.3.65:45599)
0811 22:22:58.711667 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.711671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.711845 (+ 174us) server_negotiation.cc:407] Connection header received
0811 22:22:58.712055 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.712059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.712107 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.712151 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:22:58.712565 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.713340 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.715963 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.716540 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.716627 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.716705 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.717327 (+ 622us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.717411 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260811 22:22:58.717733 20835 raft_consensus.cc:2454] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73946b78ced4491fb7759cf18360aa13 in term 0.
I20260811 22:22:58.718262 20941 leader_election.cc:302] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5a08aca64f44f486f34044a96fe38e, 73946b78ced4491fb7759cf18360aa13; no voters:
I20260811 22:22:58.718263 21193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.711001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:45599 (local address 127.29.3.67:51609)
0811 22:22:58.711751 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.711767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:22:58.711938 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:22:58.712289 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:22:58.712292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:22:58.712307 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:22:58.712452 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.712458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.714486 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.714489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.715809 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:22:58.715822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.717075 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.717078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:22:58.717181 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.717204 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:22:58.717968 (+ 764us) client_negotiation.cc:240] Negotiation successful
0811 22:22:58.718049 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":139,"threads_started":1}
I20260811 22:22:58.718745 21189 raft_consensus.cc:2781] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:22:58.718858 21189 raft_consensus.cc:482] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:22:58.718897 21189 raft_consensus.cc:3037] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:58.719270 20590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0352621bc894b4d9d8842c2936dbd61" candidate_uuid: "73946b78ced4491fb7759cf18360aa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badbb18c828843b69721aead91bdd343" is_pre_election: true
I20260811 22:22:58.719580 20590 raft_consensus.cc:2454] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73946b78ced4491fb7759cf18360aa13 in term 0.
I20260811 22:22:58.720206 21189 raft_consensus.cc:504] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.720584 21189 leader_election.cc:288] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [CANDIDATE]: Term 1 election: Requested vote from peers badbb18c828843b69721aead91bdd343 (127.29.3.65:45599), 4e5a08aca64f44f486f34044a96fe38e (127.29.3.66:38447)
I20260811 22:22:58.720978 20590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0352621bc894b4d9d8842c2936dbd61" candidate_uuid: "73946b78ced4491fb7759cf18360aa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "badbb18c828843b69721aead91bdd343"
I20260811 22:22:58.721031 20835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0352621bc894b4d9d8842c2936dbd61" candidate_uuid: "73946b78ced4491fb7759cf18360aa13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e5a08aca64f44f486f34044a96fe38e"
I20260811 22:22:58.721101 20590 raft_consensus.cc:3037] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:58.721153 20835 raft_consensus.cc:3037] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:22:58.722396 20835 raft_consensus.cc:2454] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73946b78ced4491fb7759cf18360aa13 in term 1.
I20260811 22:22:58.722539 20590 raft_consensus.cc:2454] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73946b78ced4491fb7759cf18360aa13 in term 1.
I20260811 22:22:58.722712 20941 leader_election.cc:302] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e5a08aca64f44f486f34044a96fe38e, 73946b78ced4491fb7759cf18360aa13; no voters:
I20260811 22:22:58.722888 21189 raft_consensus.cc:2781] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:22:58.723129 21189 raft_consensus.cc:686] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 1 LEADER]: Becoming Leader. State: Replica: 73946b78ced4491fb7759cf18360aa13, State: Running, Role: LEADER
I20260811 22:22:58.723541 21189 consensus_queue.cc:237] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } }
I20260811 22:22:58.727078 20342 catalog_manager.cc:5495] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73946b78ced4491fb7759cf18360aa13 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "73946b78ced4491fb7759cf18360aa13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73946b78ced4491fb7759cf18360aa13" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20260811 22:22:58.753036 21161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.747827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60064 (local address 127.29.3.67:42811)
0811 22:22:58.747955 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 22:22:58.747959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:22:58.748071 (+ 112us) server_negotiation.cc:407] Connection header received
0811 22:22:58.748245 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:22:58.748248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:22:58.748322 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:22:58.748439 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:22:58.748736 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.749486 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.750762 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:22:58.751083 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:22:58.751188 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:22:58.751644 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:22:58.752234 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:22:58.752355 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:22:58.752820 (+ 465us) server_negotiation.cc:299] Negotiation successful
0811 22:22:58.752892 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260811 22:22:58.754251 21119 tablet.cc:2369] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:22:58.762903 20835 raft_consensus.cc:1261] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 1 FOLLOWER]: Refusing update from remote peer 73946b78ced4491fb7759cf18360aa13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:22:58.762904 20590 raft_consensus.cc:1261] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 1 FOLLOWER]: Refusing update from remote peer 73946b78ced4491fb7759cf18360aa13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:22:58.763942 21196 consensus_queue.cc:1035] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e5a08aca64f44f486f34044a96fe38e" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:22:58.764446 21196 consensus_queue.cc:1035] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "badbb18c828843b69721aead91bdd343" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:22:58.774343 21199 mvcc.cc:205] Tried to move back new op lower bound from 7317450665002713088 to 7317450664854953984. Current Snapshot: MvccSnapshot[applied={T|T < 7317450665002713088}]
I20260811 22:22:58.775477 21200 mvcc.cc:205] Tried to move back new op lower bound from 7317450665002713088 to 7317450664854953984. Current Snapshot: MvccSnapshot[applied={T|T < 7317450665002713088}]
I20260811 22:22:58.783480 21204 mvcc.cc:205] Tried to move back new op lower bound from 7317450665002713088 to 7317450664854953984. Current Snapshot: MvccSnapshot[applied={T|T < 7317450665002713088}]
I20260811 22:22:58.785611 20342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:22:58.785799 20342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:22:58.788679 20342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a64c109efba4950bacb6f4ba596cc37: Sending DeleteTablet for 3 replicas of tablet c0352621bc894b4d9d8842c2936dbd61
I20260811 22:22:58.789568 21009 tablet_service.cc:1514] Processing DeleteTablet for tablet c0352621bc894b4d9d8842c2936dbd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:22:58 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:60054
I20260811 22:22:58.789728 20760 tablet_service.cc:1514] Processing DeleteTablet for tablet c0352621bc894b4d9d8842c2936dbd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:22:58 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39194
I20260811 22:22:58.790725 20530 tablet_service.cc:1514] Processing DeleteTablet for tablet c0352621bc894b4d9d8842c2936dbd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:22:58 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:39668
I20260811 22:22:58.791318 21221 tablet_replica.cc:330] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343: stopping tablet replica
I20260811 22:22:58.791785 21221 raft_consensus.cc:2229] T c0352621bc894b4d9d8842c2936dbd61 P badbb18c828843b69721aead91bdd343 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:22:58.791888 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 20398
I20260811 22:22:58.795601 21217 tablet_replica.cc:330] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: stopping tablet replica
I20260811 22:22:58.795981 21217 raft_consensus.cc:2229] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:22:58.796370 21217 raft_consensus.cc:2258] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:22:58.797863 21217 ts_tablet_manager.cc:1905] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:22:58.801080 21217 ts_tablet_manager.cc:1918] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:22:58.801186 21217 log.cc:1196] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/c0352621bc894b4d9d8842c2936dbd61
I20260811 22:22:58.801486 21217 ts_tablet_manager.cc:1939] T c0352621bc894b4d9d8842c2936dbd61 P 4e5a08aca64f44f486f34044a96fe38e: Deleting consensus metadata
Aug 11 22:22:58 dist-test-slave-4tdj krb5kdc[18560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486978, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for HTTP/127.29.3.84@KRBTEST.COM
I20260811 22:22:58.808527 20321 catalog_manager.cc:4841] TS 4e5a08aca64f44f486f34044a96fe38e (127.29.3.66:38447): tablet c0352621bc894b4d9d8842c2936dbd61 (table test-table [id=37a01ce148844581b0ffe46ada86ccb9]) successfully deleted
W20260811 22:22:58.817634 20310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:45599 (error 108)
I20260811 22:22:58.818946 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.818609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:45599 (local address 127.0.0.1:39684)
0811 22:22:58.818762 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.818840 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 22:22:58.819137 21215 tablet_replica.cc:330] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: stopping tablet replica
W20260811 22:22:58.819168 20310 catalog_manager.cc:4584] TS badbb18c828843b69721aead91bdd343 (127.29.3.65:45599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0352621bc894b4d9d8842c2936dbd61: Network error: Client connection negotiation failed: client connection to 127.29.3.65:45599: connect: Connection refused (error 111)
I20260811 22:22:58.819541 21215 raft_consensus.cc:2229] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:22:58.820009 21215 raft_consensus.cc:2258] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:22:58.820434 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 20675
I20260811 22:22:58.838021 21215 ts_tablet_manager.cc:1905] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:22:58.841567 21215 ts_tablet_manager.cc:1918] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:22:58.841811 21215 log.cc:1196] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/c0352621bc894b4d9d8842c2936dbd61
I20260811 22:22:58.842257 21215 ts_tablet_manager.cc:1939] T c0352621bc894b4d9d8842c2936dbd61 P 73946b78ced4491fb7759cf18360aa13: Deleting consensus metadata
I20260811 22:22:58.843955 20315 catalog_manager.cc:4841] TS 73946b78ced4491fb7759cf18360aa13 (127.29.3.67:42811): tablet c0352621bc894b4d9d8842c2936dbd61 (table test-table [id=37a01ce148844581b0ffe46ada86ccb9]) successfully deleted
I20260811 22:22:58.852087 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 20908
I20260811 22:22:58.872782 21163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:22:58.872339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:45599 (local address 127.0.0.1:39700)
0811 22:22:58.872613 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0811 22:22:58.872679 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:45599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260811 22:22:58.873618 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 20278
I20260811 22:22:58.900499 29709 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 22:22:59.245746 29709 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 22:22:59.245867 29709 mini_ranger.cc:66] Stopping Ranger...
I20260811 22:22:59.590950 29709 mini_ranger.cc:68] Stopped Ranger
2026-08-11 22:22:59.591 UTC [18620] LOG: received smart shutdown request
2026-08-11 22:22:59.595 UTC [18620] LOG: background worker "logical replication launcher" (PID 18635) exited with exit code 1
2026-08-11 22:22:59.596 UTC [18625] LOG: shutting down
2026-08-11 22:22:59.767 UTC [18620] LOG: database system is shut down
2026-08-11T22:22:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52442 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:22:59 dist-test-slave-4tdj krb5kdc[21240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:22:59 dist-test-slave-4tdj krb5kdc[21240](info): set up 2 sockets
Aug 11 22:22:59 dist-test-slave-4tdj krb5kdc[21240](info): commencing operation
krb5kdc: starting...
W20260811 22:23:01.084052 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:23:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:23:01.214156 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35665
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35665,127.29.3.125:35889,127.29.3.124:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:01.330608 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:01.330919 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:01.330977 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:01.335054 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:01.335125 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:01.335155 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:01.335175 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:01.335196 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:01.341092 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35665,127.29.3.125:35889,127.29.3.124:40359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:01.342355 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:01.343464 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:01.352209 21272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:01.352231 21274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:01.352391 21264 server_base.cc:1034] running on GCE node
W20260811 22:23:01.352401 21271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:01.353209 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:01.354437 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:01.355620 21264 hybrid_clock.cc:648] HybridClock initialized: now 1786486981355604 us; error 27 us; skew 500 ppm
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:01.358723 21264 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:01.359678 21264 webserver.cc:460] Webserver started at http://127.29.3.126:38991/ using document root <none> and password file <none>
I20260811 22:23:01.360250 21264 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:01.360312 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:01.360519 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:01.362725 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "e541a16d89ec4397830d0de13e68ae20"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "2a2fcd9178a4de166383d039d7f8270f"
server_key_iv: "bb9a35d4b079a3f727bf71bb895565f0"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.363332 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "e541a16d89ec4397830d0de13e68ae20"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "2a2fcd9178a4de166383d039d7f8270f"
server_key_iv: "bb9a35d4b079a3f727bf71bb895565f0"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.367410 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:23:01.369979 21287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:01.371407 21264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:23:01.371547 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "e541a16d89ec4397830d0de13e68ae20"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "2a2fcd9178a4de166383d039d7f8270f"
server_key_iv: "bb9a35d4b079a3f727bf71bb895565f0"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.371660 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:01.392518 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:01.395931 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:01.396116 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:01.403510 21264 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:35665
I20260811 22:23:01.403541 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:35665 every 8 connection(s)
I20260811 22:23:01.404567 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:01.407682 21382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:01.412161 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 21264
I20260811 22:23:01.411831 21382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.412313 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:23:01.412657 29709 external_mini_cluster.cc:1428] Setting key 0005e7bb528ef43c49a9fa13fdd20d25
WARNING: no policy specified for kudu/127.29.3.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.125@KRBTEST.COM" created.
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:01.430770 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.414633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35665 (local address 127.0.0.1:60182)
0811 22:23:01.415291 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.415333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.416213 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.416938 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.416944 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.417413 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.417740 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.417758 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.419026 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.419031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.419777 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.419787 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.420452 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.420457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.420586 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.421225 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.421249 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.425351 (+ 4102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.427496 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.427506 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.427525 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.427772 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.428177 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.428187 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.428189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.428318 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.429172 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.429186 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.429366 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.429531 (+ 165us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.429818 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"spinlock_wait_cycles":54144,"thread_start_us":208,"threads_started":1}
I20260811 22:23:01.430778 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.414634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.29.3.126:35665)
0811 22:23:01.415151 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.415174 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.415452 (+ 278us) server_negotiation.cc:407] Connection header received
0811 22:23:01.416542 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.416546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.416738 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.416922 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.417905 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.418886 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.419901 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.420288 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.420456 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.425585 (+ 5129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.425593 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.425605 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.425633 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.427372 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.427892 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.427899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.427901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.428059 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.428415 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.428418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.428420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.429032 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.429187 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.429632 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.429913 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":361,"thread_start_us":155,"threads_started":1}
I20260811 22:23:01.433178 21382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.433957 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.433639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36360)
0811 22:23:01.433809 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.433890 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260811 22:23:01.434166 21289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
I20260811 22:23:01.435122 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.434868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36368)
0811 22:23:01.434984 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.435042 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 22:23:01.435247 21382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:35889: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab.
Entry for principal kudu/127.29.3.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab.
I20260811 22:23:01.442044 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:35889
--webserver_interface=127.29.3.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab
--principal=kudu/127.29.3.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35665,127.29.3.125:35889,127.29.3.124:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:01.484511 21382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } attempt: 1
I20260811 22:23:01.485384 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.485075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36376)
0811 22:23:01.485233 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.485304 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 22:23:01.486215 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.485993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36378)
0811 22:23:01.486105 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.486153 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 22:23:01.486374 21382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:35889: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
I20260811 22:23:01.554520 21382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } attempt: 2
I20260811 22:23:01.555362 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.555068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36380)
0811 22:23:01.555219 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.555282 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 22:23:01.556314 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.556065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36394)
0811 22:23:01.556192 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.556247 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260811 22:23:01.556464 21382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:35889: Network error: Client connection negotiation failed: client connection to 127.29.3.125:35889: connect: Connection refused (error 111)
W20260811 22:23:01.561956 21391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:01.562266 21391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:01.562354 21391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:01.566552 21391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:01.566630 21391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:01.566653 21391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:01.566674 21391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:01.566707 21391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:01.572393 21391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35665,127.29.3.125:35889,127.29.3.124:40359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab
--principal=kudu/127.29.3.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:35889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.3.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:01.573604 21391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:01.574751 21391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:01.583792 21398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:01.583792 21400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:01.583855 21391 server_base.cc:1034] running on GCE node
W20260811 22:23:01.583796 21397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:01.584713 21391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:01.585925 21391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:01.587116 21391 hybrid_clock.cc:648] HybridClock initialized: now 1786486981587093 us; error 43 us; skew 500 ppm
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:01.590075 21391 init.cc:377] Logged in from keytab as kudu/127.29.3.125@KRBTEST.COM (short username kudu)
I20260811 22:23:01.591104 21391 webserver.cc:460] Webserver started at http://127.29.3.125:33359/ using document root <none> and password file <none>
I20260811 22:23:01.591707 21391 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:01.591780 21391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:01.591991 21391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:01.594099 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data/instance:
uuid: "5bbe16d9264e4d67b960d760f8c23ab7"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "637970e0f19d09a7470ebf80b220c20f"
server_key_iv: "47ec5d6e4b458bb9e4aa58890fe05590"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.594710 21391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal/instance:
uuid: "5bbe16d9264e4d67b960d760f8c23ab7"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "637970e0f19d09a7470ebf80b220c20f"
server_key_iv: "47ec5d6e4b458bb9e4aa58890fe05590"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.598507 21391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:23:01.601008 21409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:01.602238 21391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:23:01.602407 21391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal
uuid: "5bbe16d9264e4d67b960d760f8c23ab7"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "637970e0f19d09a7470ebf80b220c20f"
server_key_iv: "47ec5d6e4b458bb9e4aa58890fe05590"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.602509 21391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:01.623594 21391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:01.627007 21391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:01.627226 21391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:01.634549 21391 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.125:35889
I20260811 22:23:01.634580 21500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.125:35889 every 8 connection(s)
I20260811 22:23:01.635548 21391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
I20260811 22:23:01.638494 21502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:01.639842 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 21391
I20260811 22:23:01.639987 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-1/wal/instance
I20260811 22:23:01.640358 29709 external_mini_cluster.cc:1428] Setting key 49535acadbb7238d6d2495aa980ae825
I20260811 22:23:01.642892 21502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.647627 21382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } attempt: 3
WARNING: no policy specified for kudu/127.29.3.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.124@KRBTEST.COM" created.
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:23:01.657975 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.648115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36404)
0811 22:23:01.648313 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.648328 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.648410 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.649143 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.649146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.649337 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.649461 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.649468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.650597 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.650600 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.651317 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.651328 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.652184 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.652189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.652306 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.652768 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.652781 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.653940 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.656062 (+ 2122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.656067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.656305 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.656652 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.656657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.656733 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.657376 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.657379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.657483 (+ 104us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.657583 (+ 100us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.657780 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"spinlock_wait_cycles":1920}
I20260811 22:23:01.658564 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.644863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.29.3.126:35665)
0811 22:23:01.645046 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.645051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.645937 (+ 886us) server_negotiation.cc:407] Connection header received
0811 22:23:01.646963 (+ 1026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.646967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.647032 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.647138 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.648556 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.649089 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.650392 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.650703 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.650785 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.653824 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.653828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.653831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.653860 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.655402 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656003 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.656006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.656008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.656107 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656507 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.656510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.656512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.656733 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.656840 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.658277 (+ 1437us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.658425 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60,"spinlock_wait_cycles":3328}
I20260811 22:23:01.659180 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.645235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35665 (local address 127.0.0.1:60190)
0811 22:23:01.645761 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.645816 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.646738 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.647290 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.647297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.647683 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.648352 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.648376 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.649259 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.649262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.650247 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.650260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.650854 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.650858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.651081 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.651813 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.651831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.653627 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.655521 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.655529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.655540 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.655898 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.656233 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.656238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.656422 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.656897 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.656903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.657242 (+ 339us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.658753 (+ 1511us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.658939 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":26,"spinlock_wait_cycles":2304,"thread_start_us":131,"threads_started":1}
I20260811 22:23:01.659186 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.648234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.29.3.125:35889)
0811 22:23:01.648682 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.648694 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.648722 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:23:01.648768 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.648773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.648973 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.649142 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.649586 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.650485 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.651491 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.652013 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.652159 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.654083 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.654090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.654105 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.654130 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.655930 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656422 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.656428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.656429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.656526 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.656823 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.656826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.656827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.657260 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.657391 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.657719 (+ 328us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.658059 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":335,"spinlock_wait_cycles":2944,"thread_start_us":104,"threads_started":1}
I20260811 22:23:01.660983 21502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.661454 21382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.662148 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.661862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53880)
0811 22:23:01.662024 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.662077 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260811 22:23:01.662874 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.662654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53892)
0811 22:23:01.662780 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.662825 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260811 22:23:01.662983 21382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:40359: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:23:01.669395 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.661675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36406)
0811 22:23:01.661908 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.661922 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.662002 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.662376 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.662379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.662550 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.662679 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.662686 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.663495 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.663498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.664061 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.664071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.664511 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.664514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.664631 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.665050 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.665063 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.666230 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.667998 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.668000 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.668002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.668185 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.668489 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.668491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.668492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.668564 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.668932 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.668934 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.669045 (+ 111us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.669134 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.669250 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":15}
I20260811 22:23:01.669422 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.661912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.29.3.125:35889)
0811 22:23:01.662017 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.662020 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.662034 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:23:01.662105 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.662108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.662151 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.662231 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.662799 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.663398 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.664161 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.664425 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.664504 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.666383 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.666387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.666389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.666414 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.667903 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.668301 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.668305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.668307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.668404 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.668652 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.668656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.668657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.668835 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.668941 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.669181 (+ 240us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.669304 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":44}
I20260811 22:23:01.670205 21502 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab.
Entry for principal kudu/127.29.3.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab.
I20260811 22:23:01.670982 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.670718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53896)
0811 22:23:01.670835 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.670925 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 22:23:01.671566 21413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
I20260811 22:23:01.672464 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.672231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53906)
0811 22:23:01.672366 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.672413 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260811 22:23:01.672619 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:40359: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
I20260811 22:23:01.672662 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:40359
--webserver_interface=127.29.3.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab
--principal=kudu/127.29.3.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35665,127.29.3.125:35889,127.29.3.124:40359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:01.694151 21382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } attempt: 1
I20260811 22:23:01.695044 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.694747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53918)
0811 22:23:01.694894 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.694958 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 22:23:01.695808 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.695600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53928)
0811 22:23:01.695700 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.695747 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 22:23:01.695932 21382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:40359: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
I20260811 22:23:01.721911 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } attempt: 1
I20260811 22:23:01.722918 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.722567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53932)
0811 22:23:01.722760 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.722830 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 22:23:01.723664 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.723436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53936)
0811 22:23:01.723552 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.723603 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 22:23:01.723788 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:40359: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
I20260811 22:23:01.771117 21382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } attempt: 2
I20260811 22:23:01.772140 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.771797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53944)
0811 22:23:01.771968 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.772035 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 22:23:01.772841 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.772652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53954)
0811 22:23:01.772739 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.772783 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260811 22:23:01.772959 21382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:40359: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
W20260811 22:23:01.790467 21516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:01.790849 21516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:01.790946 21516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 22:23:01.791926 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } attempt: 2
I20260811 22:23:01.792775 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.792477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53968)
0811 22:23:01.792627 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.792693 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 22:23:01.793471 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.793261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53978)
0811 22:23:01.793367 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.793413 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 22:23:01.793586 21502 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:40359: Network error: Client connection negotiation failed: client connection to 127.29.3.124:40359: connect: Connection refused (error 111)
W20260811 22:23:01.795504 21516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:01.795568 21516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:01.795598 21516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:01.795630 21516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:01.795660 21516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:01.801282 21516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35665,127.29.3.125:35889,127.29.3.124:40359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab
--principal=kudu/127.29.3.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:40359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.3.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:01.802604 21516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:01.803689 21516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:01.812678 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:01.812678 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:01.812700 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:01.813043 21516 server_base.cc:1034] running on GCE node
I20260811 22:23:01.813704 21516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:01.814936 21516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:01.816102 21516 hybrid_clock.cc:648] HybridClock initialized: now 1786486981816087 us; error 29 us; skew 500 ppm
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:01.818910 21516 init.cc:377] Logged in from keytab as kudu/127.29.3.124@KRBTEST.COM (short username kudu)
I20260811 22:23:01.819854 21516 webserver.cc:460] Webserver started at http://127.29.3.124:35565/ using document root <none> and password file <none>
I20260811 22:23:01.820420 21516 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:01.820487 21516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:01.820686 21516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:01.822871 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data/instance:
uuid: "c86b6b637d20484cbd2e356381bf9f6b"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "0bcc32741503d71a1b084c76906391de"
server_key_iv: "41c7cdbbfef0a4b68d7ec548bae7536f"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.823446 21516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal/instance:
uuid: "c86b6b637d20484cbd2e356381bf9f6b"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "0bcc32741503d71a1b084c76906391de"
server_key_iv: "41c7cdbbfef0a4b68d7ec548bae7536f"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.827468 21516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:23:01.829996 21535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:01.831313 21516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:23:01.831442 21516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal
uuid: "c86b6b637d20484cbd2e356381bf9f6b"
format_stamp: "Formatted at 2026-08-11 22:23:01 on dist-test-slave-4tdj"
server_key: "0bcc32741503d71a1b084c76906391de"
server_key_iv: "41c7cdbbfef0a4b68d7ec548bae7536f"
server_key_version: "encryptionkey@0"
I20260811 22:23:01.831549 21516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:01.844269 21516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:01.847391 21516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:01.847585 21516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:01.855266 21516 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.124:40359
I20260811 22:23:01.855281 21616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.124:40359 every 8 connection(s)
I20260811 22:23:01.856303 21516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
I20260811 22:23:01.859278 21617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:01.860167 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 21516
I20260811 22:23:01.860298 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786486810933177-29709-0/minicluster-data/master-2/wal/instance
I20260811 22:23:01.860625 29709 external_mini_cluster.cc:1428] Setting key 21e6185e3f29fd303122665cba49bbf4
I20260811 22:23:01.863729 21617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:01.872128 21382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } attempt: 3
I20260811 22:23:01.875952 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.861999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.29.3.124:40359)
0811 22:23:01.862460 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.862480 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.862525 (+ 45us) server_negotiation.cc:407] Connection header received
0811 22:23:01.863214 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.863232 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.863568 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.863843 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.864733 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.866012 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.867304 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.867780 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.867905 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.870653 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.870660 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.870674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.870700 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.872261 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.872723 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.872727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.872731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.872827 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.873211 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.873214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.873216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.873759 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.873967 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.874240 (+ 273us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.874652 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":311,"thread_start_us":127,"threads_started":1}
I20260811 22:23:01.875973 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.865613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35665 (local address 127.0.0.1:60204)
0811 22:23:01.865971 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.866016 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.866131 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.866533 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.866539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.866974 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.867136 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.867144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.867946 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.867950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.868672 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.868683 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.869286 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.869288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.869384 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.869885 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.869905 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.871566 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.873706 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.873715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.873721 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.873995 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.874341 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.874343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.874345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.874432 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.874903 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.874909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.875043 (+ 134us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.875175 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.875360 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":48,"thread_start_us":118,"threads_started":1}
I20260811 22:23:01.876436 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.865665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.29.3.126:35665)
0811 22:23:01.865782 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.865787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.866095 (+ 308us) server_negotiation.cc:407] Connection header received
0811 22:23:01.866330 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.866333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.866385 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.866451 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.867305 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.867808 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.868795 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.869142 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.869253 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.871703 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.871706 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.871708 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.871733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.873551 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.874111 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.874114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.874116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.874202 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.874550 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.874553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.874555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.874787 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.876099 (+ 1312us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.876180 (+ 81us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.876292 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:23:01.877861 21617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.880992 21387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.872554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53992)
0811 22:23:01.872685 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.872699 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.872775 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.873190 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.873193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.873378 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.873458 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.873463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.874314 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.874318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.874983 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.874994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.875449 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.875453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.875542 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.875934 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.875944 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.877300 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.879449 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.879452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.879454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.879717 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.880030 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.880033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.880035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.880126 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.880531 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.880534 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.880625 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.880702 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.880830 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 22:23:01.881624 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.872702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.29.3.124:40359)
0811 22:23:01.872998 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.873016 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.873027 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:23:01.873051 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.873053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.873084 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.873153 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.873569 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.874170 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.875115 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.875351 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.875428 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.877508 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.877511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.877513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.877534 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.879359 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.879813 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.879816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.879818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.879918 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.880234 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.880238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.880239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.880433 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.881263 (+ 830us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.881333 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.881441 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:01.884740 21502 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } attempt: 3
I20260811 22:23:01.885255 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: Bootstrap starting.
I20260811 22:23:01.887445 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.878715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:35889 (local address 127.0.0.1:36408)
0811 22:23:01.878840 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.878855 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.878929 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.879557 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.879559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.879677 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.879752 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.879756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.880715 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.880718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.881431 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.881440 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.881987 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.881991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.882077 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.882501 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.882516 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.884000 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.885771 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.885774 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.885777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.885996 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.886396 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.886399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.886400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.886494 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.886972 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.886975 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.887066 (+ 91us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.887168 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.887297 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260811 22:23:01.888140 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.879135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.29.3.125:35889)
0811 22:23:01.879261 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.879265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.879280 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:23:01.879323 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.879326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.879368 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.879437 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.879879 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.880597 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.881572 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.881894 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.881993 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.884129 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.884133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.884135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.884162 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.885646 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.886126 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.886130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.886131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.886228 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.886606 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.886610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.886611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.886839 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.886927 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.887876 (+ 949us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.887989 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:23:01.888330 21382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:01.889110 21617 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:23:01.889376 21382 log.cc:824] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:01.889990 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.878958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.29.3.126:35665)
0811 22:23:01.879082 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.879086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.879124 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:23:01.879157 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.879160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.879197 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.879269 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.879970 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.880446 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.881234 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.881489 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.881576 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.884301 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.884305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.884307 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.884331 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.886221 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.886743 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.886746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.886748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.886831 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.887118 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.887122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.887124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.887323 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.887485 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.888136 (+ 651us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.888246 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:23:01.893464 21382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: No bootstrap required, opened a new log
I20260811 22:23:01.894557 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.885138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:53998)
0811 22:23:01.885332 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.885347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.885419 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.886416 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.886419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.886585 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.886696 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.886703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.887727 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.887730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.888462 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.888473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.889155 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.889159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.889244 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.889675 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.889689 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.891137 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.892775 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.892778 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.892780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.893000 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.893306 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.893309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.893311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.893397 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.894078 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.894081 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.894174 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.894249 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.894405 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:23:01.895787 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.885620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.29.3.124:40359)
0811 22:23:01.886175 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.886179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.886190 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:23:01.886218 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.886220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.886259 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.886796 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.886881 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.887608 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.888746 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.889042 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.889133 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.891269 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.891273 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.891275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.891298 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.892658 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.893116 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.893119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.893121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.893204 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.893734 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.893737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.893739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.893963 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.895014 (+ 1051us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.895088 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.895429 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"spinlock_wait_cycles":768}
Aug 11 22:23:01 dist-test-slave-4tdj krb5kdc[21240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:23:01.897704 21382 raft_consensus.cc:348] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:01.898025 21382 raft_consensus.cc:374] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:01.898135 21382 raft_consensus.cc:729] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e541a16d89ec4397830d0de13e68ae20, State: Initialized, Role: FOLLOWER
I20260811 22:23:01.898175 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.889581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:40359 (local address 127.0.0.1:54002)
0811 22:23:01.889782 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:01.889794 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:01.889863 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:01.890092 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:01.890095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:01.890244 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:01.890364 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.890371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.891315 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.891319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.892085 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:01.892098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.893306 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.893309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:01.893373 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.893738 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:01.893770 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:01.895142 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:01.896782 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.896785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.896787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.896994 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.897250 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:01.897253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:01.897254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:01.897331 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:01.897719 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:01.897721 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:01.897801 (+ 80us) client_negotiation.cc:769] Sending connection context
0811 22:23:01.897897 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 22:23:01.898014 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"spinlock_wait_cycles":12544}
I20260811 22:23:01.898599 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.889773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54002 (local address 127.29.3.124:40359)
0811 22:23:01.889897 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.889901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.889915 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:23:01.889948 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.889951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.889986 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.890457 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.890537 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.891230 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.892766 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.893131 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.893217 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.895268 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.895271 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.895273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.895293 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.896677 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.897082 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.897084 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.897086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.897164 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.897413 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.897416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.897417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.897598 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.897687 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.898378 (+ 691us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.898483 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:23:01.898903 21382 consensus_queue.cc:260] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:01.899557 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:01.899750 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:01.900034 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: Bootstrap starting.
I20260811 22:23:01.900502 21382 sys_catalog.cc:564] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:01.901314 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.891176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.29.3.125:35889)
0811 22:23:01.891332 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.891336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.891347 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:23:01.891376 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.891378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.891414 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.891485 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.892686 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.893359 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.894415 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.894762 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.894835 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.897750 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.897754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.897756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.897778 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.899581 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.900039 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.900042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.900044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.900132 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.900423 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.900426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.900428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.900641 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.900764 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.901060 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.901171 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 22:23:01.904412 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: Bootstrap starting.
I20260811 22:23:01.905499 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260811 22:23:01.905891 21646 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:23:01.905958 21646 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:23:01.908576 21502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:01.909808 21617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:01.910822 21617 log.cc:824] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:01.908452 21264 master_runner.cc:186] Error getting master registration for 127.29.3.125:35889: OK, instance_id { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" instance_seqno: 1786486981632471 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:23:01.911271 21264 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:23:01.911788 21502 log.cc:824] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:01.919695 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.908157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.29.3.125:35889)
0811 22:23:01.908248 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.908252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.910808 (+ 2556us) server_negotiation.cc:407] Connection header received
0811 22:23:01.911063 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.911066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.911126 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.911179 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.912058 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.912811 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.913955 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.914356 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.914471 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.916122 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.916126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.916129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.916157 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.917791 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.918330 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.918333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.918335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.918424 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.918705 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.918708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.918710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.918937 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.919036 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.919388 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.919515 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 22:23:01.919921 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.908085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.29.3.126:35665)
0811 22:23:01.908195 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.908200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.910610 (+ 2410us) server_negotiation.cc:407] Connection header received
0811 22:23:01.912035 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.912039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.912090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.912156 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.912990 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.913758 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.914640 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.914903 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.914964 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.916427 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.916430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.916433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.916461 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.917754 (+ 1293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.918178 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.918182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.918184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.918275 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.918979 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.918983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.918984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.919212 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.919326 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.919658 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.919779 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 22:23:01.920495 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: No bootstrap required, opened a new log
I20260811 22:23:01.924929 21502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:01.925282 21502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:01.925362 21502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbe16d9264e4d67b960d760f8c23ab7, State: Initialized, Role: FOLLOWER
I20260811 22:23:01.925889 21502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:01.926468 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:01.926693 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:01.927469 21502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:01.928069 21621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:01.918754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.29.3.124:40359)
0811 22:23:01.918884 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:01.918888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:01.920446 (+ 1558us) server_negotiation.cc:407] Connection header received
0811 22:23:01.920652 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:01.920656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:01.920713 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:01.920766 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:01.921498 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.922207 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.923191 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:01.923409 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:01.923495 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:01.924779 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:01.924783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:01.924786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:01.924813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:01.926387 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.926879 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.926883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.926885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.926981 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:01.927256 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:01.927260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:01.927262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:01.927451 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:01.927562 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:01.927798 (+ 236us) server_negotiation.cc:299] Negotiation successful
0811 22:23:01.927914 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:23:01.932876 21617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: No bootstrap required, opened a new log
W20260811 22:23:01.933774 21681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:23:01.933912 21681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:23:01.936676 21617 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:01.936940 21617 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:01.936995 21617 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86b6b637d20484cbd2e356381bf9f6b, State: Initialized, Role: FOLLOWER
I20260811 22:23:01.937466 21617 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:01.938978 21617 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:01.944938 21391 master_runner.cc:263] No leader master found from master 5bbe16d9264e4d67b960d760f8c23ab7
I20260811 22:23:01.945039 21391 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:23:01.946473 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:01.946630 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:01.948935 21516 master_runner.cc:263] No leader master found from master c86b6b637d20484cbd2e356381bf9f6b
I20260811 22:23:01.949011 21516 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 22:23:01.950580 21708 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:23:01.950670 21708 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260811 22:23:02.099045 21544 tablet.cc:2369] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:23:02.128854 21418 tablet.cc:2369] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:23:02.145313 21626 raft_consensus.cc:482] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:02.145525 21626 raft_consensus.cc:504] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:02.146552 21626 leader_election.cc:288] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5bbe16d9264e4d67b960d760f8c23ab7 (127.29.3.125:35889), c86b6b637d20484cbd2e356381bf9f6b (127.29.3.124:40359)
I20260811 22:23:02.147154 21453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e541a16d89ec4397830d0de13e68ae20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" is_pre_election: true
I20260811 22:23:02.147182 21573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e541a16d89ec4397830d0de13e68ae20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86b6b637d20484cbd2e356381bf9f6b" is_pre_election: true
I20260811 22:23:02.147471 21453 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e541a16d89ec4397830d0de13e68ae20 in term 0.
I20260811 22:23:02.147501 21573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e541a16d89ec4397830d0de13e68ae20 in term 0.
I20260811 22:23:02.147918 21289 leader_election.cc:302] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bbe16d9264e4d67b960d760f8c23ab7, e541a16d89ec4397830d0de13e68ae20; no voters:
I20260811 22:23:02.148214 21626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:23:02.148309 21626 raft_consensus.cc:482] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:23:02.148358 21626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 0 FOLLOWER]: Advancing to term 1
W20260811 22:23:02.148484 21299 tablet.cc:2369] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:23:02.149529 21626 raft_consensus.cc:504] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:02.149893 21626 leader_election.cc:288] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [CANDIDATE]: Term 1 election: Requested vote from peers 5bbe16d9264e4d67b960d760f8c23ab7 (127.29.3.125:35889), c86b6b637d20484cbd2e356381bf9f6b (127.29.3.124:40359)
I20260811 22:23:02.150281 21453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e541a16d89ec4397830d0de13e68ae20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbe16d9264e4d67b960d760f8c23ab7"
I20260811 22:23:02.150352 21573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e541a16d89ec4397830d0de13e68ae20" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86b6b637d20484cbd2e356381bf9f6b"
I20260811 22:23:02.150477 21573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:02.150461 21453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:02.151813 21573 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e541a16d89ec4397830d0de13e68ae20 in term 1.
I20260811 22:23:02.151813 21453 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e541a16d89ec4397830d0de13e68ae20 in term 1.
I20260811 22:23:02.152137 21289 leader_election.cc:302] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5bbe16d9264e4d67b960d760f8c23ab7, c86b6b637d20484cbd2e356381bf9f6b, e541a16d89ec4397830d0de13e68ae20; no voters:
I20260811 22:23:02.152323 21626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:02.152557 21626 raft_consensus.cc:686] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [term 1 LEADER]: Becoming Leader. State: Replica: e541a16d89ec4397830d0de13e68ae20, State: Running, Role: LEADER
I20260811 22:23:02.152910 21626 consensus_queue.cc:237] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } }
I20260811 22:23:02.154783 21710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e541a16d89ec4397830d0de13e68ae20. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.154892 21710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:02.155246 21713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:02.156162 21713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:02.160740 21453 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [term 1 FOLLOWER]: Refusing update from remote peer e541a16d89ec4397830d0de13e68ae20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:02.160785 21573 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [term 1 FOLLOWER]: Refusing update from remote peer e541a16d89ec4397830d0de13e68ae20: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:02.161386 21710 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:02.161547 21626 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:02.165251 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e541a16d89ec4397830d0de13e68ae20. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.165380 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:02.165343 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e541a16d89ec4397830d0de13e68ae20. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.165477 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:02.168255 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.168370 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:02.168466 21664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.168573 21664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:02.168596 21626 sys_catalog.cc:455] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.168687 21626 sys_catalog.cc:458] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:02.169365 21684 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e541a16d89ec4397830d0de13e68ae20" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e541a16d89ec4397830d0de13e68ae20" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35665 } } peers { permanent_uuid: "5bbe16d9264e4d67b960d760f8c23ab7" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 35889 } } peers { permanent_uuid: "c86b6b637d20484cbd2e356381bf9f6b" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 40359 } } }
I20260811 22:23:02.169490 21684 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:23:02.169967 21713 catalog_manager.cc:1347] Generated new cluster ID: afeffb0c638e44cb886003c6590873e2
I20260811 22:23:02.170028 21713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:23:02.184741 21713 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:23:02.185892 21713 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:23:02.201368 21713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e541a16d89ec4397830d0de13e68ae20: Generated new TSK 0
I20260811 22:23:02.202512 21713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:23:02.935549 21681 catalog_manager.cc:1259] Loaded cluster ID: afeffb0c638e44cb886003c6590873e2
I20260811 22:23:02.935642 21681 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: loading cluster ID for follower catalog manager: success
I20260811 22:23:02.937450 21681 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: acquiring CA information for follower catalog manager: success
I20260811 22:23:02.939056 21681 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5bbe16d9264e4d67b960d760f8c23ab7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:23:02.952121 21708 catalog_manager.cc:1259] Loaded cluster ID: afeffb0c638e44cb886003c6590873e2
I20260811 22:23:02.952204 21708 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: loading cluster ID for follower catalog manager: success
I20260811 22:23:02.953917 21708 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: acquiring CA information for follower catalog manager: success
I20260811 22:23:02.955233 21708 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c86b6b637d20484cbd2e356381bf9f6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:23:03.275571 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 21264
I20260811 22:23:03.288034 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 21391
I20260811 22:23:03.303997 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 21516
2026-08-11T22:23:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3428 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:23:03 dist-test-slave-4tdj krb5kdc[21732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:23:03 dist-test-slave-4tdj krb5kdc[21732](info): set up 2 sockets
Aug 11 22:23:03 dist-test-slave-4tdj krb5kdc[21732](info): commencing operation
krb5kdc: starting...
W20260811 22:23:04.495545 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:23:04 dist-test-slave-4tdj krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:23:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:23:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:23:04.625594 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44539
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:46827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:44539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:04.740818 21753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:04.741130 21753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:04.741173 21753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:04.745394 21753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:04.745482 21753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:04.745500 21753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:04.745517 21753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:04.745532 21753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:04.751185 21753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:46827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:44539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:04.752410 21753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:04.753502 21753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:04.762537 21765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:04.762723 21753 server_base.cc:1034] running on GCE node
W20260811 22:23:04.762528 21759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:04.762528 21761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:04.763489 21753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:04.764627 21753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:04.765794 21753 hybrid_clock.cc:648] HybridClock initialized: now 1786486984765777 us; error 30 us; skew 500 ppm
Aug 11 22:23:04 dist-test-slave-4tdj krb5kdc[21732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486984, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:04.773288 21753 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:04.774356 21753 webserver.cc:460] Webserver started at http://127.29.3.126:40879/ using document root <none> and password file <none>
I20260811 22:23:04.774953 21753 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:04.775017 21753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:04.775210 21753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:04.777374 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "6deeced80acf4b6d93d6fae4ee28a7cd"
format_stamp: "Formatted at 2026-08-11 22:23:04 on dist-test-slave-4tdj"
server_key: "bfb7cbf8d295f6ba00e1cee58b40bba4"
server_key_iv: "126a8a8c87d21edf909f63906c9c9643"
server_key_version: "encryptionkey@0"
I20260811 22:23:04.777894 21753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "6deeced80acf4b6d93d6fae4ee28a7cd"
format_stamp: "Formatted at 2026-08-11 22:23:04 on dist-test-slave-4tdj"
server_key: "bfb7cbf8d295f6ba00e1cee58b40bba4"
server_key_iv: "126a8a8c87d21edf909f63906c9c9643"
server_key_version: "encryptionkey@0"
I20260811 22:23:04.781827 21753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260811 22:23:04.784428 21787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:04.785768 21753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 22:23:04.785903 21753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "6deeced80acf4b6d93d6fae4ee28a7cd"
format_stamp: "Formatted at 2026-08-11 22:23:04 on dist-test-slave-4tdj"
server_key: "bfb7cbf8d295f6ba00e1cee58b40bba4"
server_key_iv: "126a8a8c87d21edf909f63906c9c9643"
server_key_version: "encryptionkey@0"
I20260811 22:23:04.786028 21753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:04.809649 21753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:04.812812 21753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:04.813007 21753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:04.820415 21753 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:44539
I20260811 22:23:04.820431 21864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:44539 every 8 connection(s)
I20260811 22:23:04.821444 21753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:04.823523 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 21753
I20260811 22:23:04.823661 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:23:04.823975 29709 external_mini_cluster.cc:1428] Setting key 959de1d2f8bfdc902acbe4cfa16a918e
I20260811 22:23:04.824841 21865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:04.831825 21865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd: Bootstrap starting.
Aug 11 22:23:04 dist-test-slave-4tdj krb5kdc[21732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:04.835002 21865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:04.835783 21865 log.cc:824] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:04.838234 21865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd: No bootstrap required, opened a new log
I20260811 22:23:04.839632 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:04.825387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.29.3.126:44539)
0811 22:23:04.825812 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:04.825829 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:04.825867 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:23:04.826577 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:04.826596 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:04.826941 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:04.827241 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:04.828163 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:04.829209 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:04.830424 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:04.830973 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:04.831110 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:04.834481 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:04.834488 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:04.834506 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:04.834546 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:04.836528 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:04.837059 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:04.837064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:04.837070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:04.837205 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:04.837533 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:04.837535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:04.837537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:04.837982 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:04.838143 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:04.838530 (+ 387us) server_negotiation.cc:299] Negotiation successful
0811 22:23:04.838853 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":129,"threads_started":1}
I20260811 22:23:04.842438 21865 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44539 } }
I20260811 22:23:04.842774 21865 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:04.842876 21865 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6deeced80acf4b6d93d6fae4ee28a7cd, State: Initialized, Role: FOLLOWER
I20260811 22:23:04.842981 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:23:04.843364 21865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44539 } }
I20260811 22:23:04.843514 21865 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:23:04.843580 21865 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:23:04.843690 21865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:04.844934 21865 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44539 } }
I20260811 22:23:04.845283 21865 leader_election.cc:302] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6deeced80acf4b6d93d6fae4ee28a7cd; no voters:
I20260811 22:23:04.845573 21865 leader_election.cc:288] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:23:04.845757 21872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:04.846040 21872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [term 1 LEADER]: Becoming Leader. State: Replica: 6deeced80acf4b6d93d6fae4ee28a7cd, State: Running, Role: LEADER
I20260811 22:23:04.846412 21872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44539 } }
I20260811 22:23:04.846961 21865 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:04.848133 21875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6deeced80acf4b6d93d6fae4ee28a7cd. Latest consensus state: current_term: 1 leader_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44539 } } }
I20260811 22:23:04.848282 21875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:04.848675 21873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6deeced80acf4b6d93d6fae4ee28a7cd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44539 } } }
I20260811 22:23:04.848842 21873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:04.848726 21886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:04.851962 21886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:04.858671 21886 catalog_manager.cc:1347] Generated new cluster ID: 52b3f85a3ef74f3ca162560038c5c46f
I20260811 22:23:04.858767 21886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:23:04.868464 21886 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:23:04.869861 21886 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:23:04.882129 21886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6deeced80acf4b6d93d6fae4ee28a7cd: Generated new TSK 0
I20260811 22:23:04.883149 21886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:23:04.898993 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 21753
2026-08-11T22:23:04Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1597 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:23:04 dist-test-slave-4tdj krb5kdc[21907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:23:04 dist-test-slave-4tdj krb5kdc[21907](info): set up 2 sockets
Aug 11 22:23:04 dist-test-slave-4tdj krb5kdc[21907](info): commencing operation
krb5kdc: starting...
W20260811 22:23:06.096786 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:23:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:23:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:23:06.228956 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39005
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:06.344089 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:06.344384 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:06.344444 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:06.348584 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:06.348671 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:06.348698 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:06.348719 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:06.348738 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:06.354369 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:06.355573 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:06.356649 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:06.365382 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:06.365443 21933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:06.365382 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:06.365724 21926 server_base.cc:1034] running on GCE node
I20260811 22:23:06.366325 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:06.367470 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:06.368669 21926 hybrid_clock.cc:648] HybridClock initialized: now 1786486986368654 us; error 46 us; skew 500 ppm
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:06.371567 21926 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:06.372586 21926 webserver.cc:460] Webserver started at http://127.29.3.126:39965/ using document root <none> and password file <none>
I20260811 22:23:06.373163 21926 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:06.373235 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:06.373450 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:06.375604 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "cce39c1cd1104ed3a181790c0eae3276"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "2b0a93c501096d2982e8a5c77b7da961"
server_key_iv: "97dedf1f312e9d8b959ebf24e58f2bb2"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.376186 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "cce39c1cd1104ed3a181790c0eae3276"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "2b0a93c501096d2982e8a5c77b7da961"
server_key_iv: "97dedf1f312e9d8b959ebf24e58f2bb2"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.380098 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:23:06.382738 21953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.384099 21926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:06.384238 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "cce39c1cd1104ed3a181790c0eae3276"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "2b0a93c501096d2982e8a5c77b7da961"
server_key_iv: "97dedf1f312e9d8b959ebf24e58f2bb2"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.384346 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:06.399171 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:06.402259 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:06.402485 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:06.409571 21926 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:39005
I20260811 22:23:06.409898 22022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:39005 every 8 connection(s)
I20260811 22:23:06.410626 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:06.413631 22023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:06.416612 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 21926
I20260811 22:23:06.416764 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:23:06.417114 29709 external_mini_cluster.cc:1428] Setting key 0120b9ef2b234703a8c28fed5157834b
I20260811 22:23:06.420246 22023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap starting.
I20260811 22:23:06.423079 22023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:06.424064 22023 log.cc:824] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:06.426627 22023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: No bootstrap required, opened a new log
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:06.430285 22023 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:06.430590 22023 raft_consensus.cc:374] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:06.430644 22023 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce39c1cd1104ed3a181790c0eae3276, State: Initialized, Role: FOLLOWER
I20260811 22:23:06.431161 22023 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:06.431308 22023 raft_consensus.cc:388] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:23:06.431380 22023 raft_consensus.cc:482] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:23:06.431486 22023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:06.432108 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:06.418702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.29.3.126:39005)
0811 22:23:06.419168 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:06.419185 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:06.419223 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:23:06.419889 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:06.419908 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:06.420269 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:06.420622 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:06.421513 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.422641 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.423838 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.424366 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.424502 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:06.427248 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:06.427254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:06.427270 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:06.427300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:06.428962 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:06.429394 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:06.429399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:06.429405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:06.429525 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:06.429794 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:06.429797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:06.429798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:06.430272 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:06.430515 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:06.430832 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 22:23:06.431180 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":128,"threads_started":1}
I20260811 22:23:06.432559 22023 raft_consensus.cc:504] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:06.432899 22023 leader_election.cc:302] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cce39c1cd1104ed3a181790c0eae3276; no voters:
I20260811 22:23:06.433235 22023 leader_election.cc:288] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:23:06.433327 22031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:06.433549 22031 raft_consensus.cc:686] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 LEADER]: Becoming Leader. State: Replica: cce39c1cd1104ed3a181790c0eae3276, State: Running, Role: LEADER
I20260811 22:23:06.433914 22031 consensus_queue.cc:237] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:06.434615 22023 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:06.435194 22031 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cce39c1cd1104ed3a181790c0eae3276. Latest consensus state: current_term: 1 leader_uuid: "cce39c1cd1104ed3a181790c0eae3276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } } }
I20260811 22:23:06.435369 22031 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:06.435701 22032 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cce39c1cd1104ed3a181790c0eae3276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } } }
I20260811 22:23:06.435827 22032 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:06.435866 22039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:06.439225 22039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:06.445284 22039 catalog_manager.cc:1347] Generated new cluster ID: 6733f95e016d4ccc8ca7c87201751c46
I20260811 22:23:06.445381 22039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:23:06.461891 22039 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:23:06.463218 22039 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:23:06.469835 22039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Generated new TSK 0
I20260811 22:23:06.470769 22039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:23:06.514588 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39005
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:06.643071 22064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:06.643362 22064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:06.643406 22064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:06.647531 22064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:06.647614 22064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:06.647676 22064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:23:06.653141 22064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:06.654330 22064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:06.655405 22064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:06.665336 22077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:06.665390 22073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:06.665390 22074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:06.665680 22064 server_base.cc:1034] running on GCE node
I20260811 22:23:06.666217 22064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:06.666931 22064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:06.668107 22064 hybrid_clock.cc:648] HybridClock initialized: now 1786486986668085 us; error 47 us; skew 500 ppm
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:06.670977 22064 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:23:06.671903 22064 webserver.cc:460] Webserver started at http://127.29.3.65:33841/ using document root <none> and password file <none>
I20260811 22:23:06.672501 22064 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:06.672569 22064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:06.672776 22064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:06.675004 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "861058085af44412b85f3256353ea8b4"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "f97c18368d8681e647f760a16f7a802d"
server_key_iv: "819d087d4dc4869d819c307693fa6906"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.675575 22064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "861058085af44412b85f3256353ea8b4"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "f97c18368d8681e647f760a16f7a802d"
server_key_iv: "819d087d4dc4869d819c307693fa6906"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.679484 22064 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:23:06.682005 22085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.683430 22064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:06.683568 22064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "861058085af44412b85f3256353ea8b4"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "f97c18368d8681e647f760a16f7a802d"
server_key_iv: "819d087d4dc4869d819c307693fa6906"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.683691 22064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:06.698588 22064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:06.701498 22064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:06.701691 22064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:06.702345 22064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:06.703249 22064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:06.703298 22064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.703353 22064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:06.703377 22064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.712335 22064 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:44655
I20260811 22:23:06.712365 22252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:44655 every 8 connection(s)
I20260811 22:23:06.713369 22064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:06.723320 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 22064
I20260811 22:23:06.723441 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:23:06.723724 29709 external_mini_cluster.cc:1428] Setting key d356321ca7acabcc6ddd4a8b4550aa07
I20260811 22:23:06.727494 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:06.715321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:40849 (local address 127.29.3.126:39005)
0811 22:23:06.715496 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:06.715501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:06.716354 (+ 853us) server_negotiation.cc:407] Connection header received
0811 22:23:06.717682 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:06.717685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:06.717736 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:06.717812 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:06.719333 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.719892 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.721066 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.721279 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.721351 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:06.723995 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:06.723998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:06.724000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:06.724028 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:06.725404 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:06.725908 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:06.725912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:06.725914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:06.725997 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:06.726355 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:06.726359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:06.726360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:06.726554 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:06.726643 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:06.727185 (+ 542us) server_negotiation.cc:299] Negotiation successful
0811 22:23:06.727322 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:23:06.728430 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:06.715731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:40849)
0811 22:23:06.716195 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:06.716241 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:06.717468 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:06.717971 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:06.717978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:06.718585 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:06.719172 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:06.719192 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.720019 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.720022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:06.720968 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:06.720978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.721444 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.721448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:06.721605 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:06.722191 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:06.722208 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:06.723812 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:06.725516 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:06.725522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:06.725540 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:06.725804 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:06.726080 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:06.726083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:06.726085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:06.726247 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:06.726657 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:06.726665 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:06.726932 (+ 267us) client_negotiation.cc:769] Sending connection context
0811 22:23:06.727105 (+ 173us) client_negotiation.cc:240] Negotiation successful
0811 22:23:06.727481 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":121,"threads_started":1}
I20260811 22:23:06.729673 22253 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39005
I20260811 22:23:06.729976 22253 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:06.730690 22253 heartbeater.cc:511] Master 127.29.3.126:39005 requested a full tablet report, sending...
I20260811 22:23:06.732451 21973 ts_manager.cc:194] Registered new tserver with Master: 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655)
I20260811 22:23:06.733629 21973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:40849
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:23:06.754043 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39005
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:06.880247 22260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:06.880534 22260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:06.880586 22260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:06.885268 22260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:06.885339 22260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:06.885416 22260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:23:06.891315 22260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:06.892468 22260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:06.893579 22260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:06.903926 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:06.903972 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:06.903924 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:06.904248 22260 server_base.cc:1034] running on GCE node
I20260811 22:23:06.904776 22260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:06.905509 22260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:06.906682 22260 hybrid_clock.cc:648] HybridClock initialized: now 1786486986906650 us; error 46 us; skew 500 ppm
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:06.909515 22260 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:23:06.910727 22260 webserver.cc:460] Webserver started at http://127.29.3.66:37117/ using document root <none> and password file <none>
I20260811 22:23:06.911281 22260 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:06.911345 22260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:06.911556 22260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:06.913763 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "f358c50f28684a878bd82a66f0498a39"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "dda47d301bad079ad70db636d6a0698d"
server_key_iv: "c18e8a0765cce04b8e8e2b60c9337201"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.914404 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "f358c50f28684a878bd82a66f0498a39"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "dda47d301bad079ad70db636d6a0698d"
server_key_iv: "c18e8a0765cce04b8e8e2b60c9337201"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.918488 22260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:23:06.921319 22288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.922866 22260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:06.922994 22260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "f358c50f28684a878bd82a66f0498a39"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "dda47d301bad079ad70db636d6a0698d"
server_key_iv: "c18e8a0765cce04b8e8e2b60c9337201"
server_key_version: "encryptionkey@0"
I20260811 22:23:06.923132 22260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:06.943059 22260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:06.946408 22260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:06.946599 22260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:06.947219 22260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:06.948079 22260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:06.948137 22260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.948199 22260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:06.948225 22260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:06.957473 22260 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:39279
I20260811 22:23:06.957497 22457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:39279 every 8 connection(s)
I20260811 22:23:06.958496 22260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:23:06.964742 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 22260
I20260811 22:23:06.964876 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:23:06.965153 29709 external_mini_cluster.cc:1428] Setting key f78e571a31872db0fd279c1cfc8a43a7
Aug 11 22:23:06 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486986, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:06.973439 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:06.960414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:56243 (local address 127.29.3.126:39005)
0811 22:23:06.960570 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:06.960574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:06.961518 (+ 944us) server_negotiation.cc:407] Connection header received
0811 22:23:06.963102 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:06.963105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:06.963166 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:06.963240 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:06.964583 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.965163 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.966506 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.966809 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.966911 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:06.969723 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:06.969728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:06.969731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:06.969764 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:06.971248 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:06.971817 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:06.971820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:06.971821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:06.971911 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:06.972280 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:06.972283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:06.972285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:06.972486 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:06.972578 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:06.973129 (+ 551us) server_negotiation.cc:299] Negotiation successful
0811 22:23:06.973264 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:23:06.974380 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:06.960806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:56243)
0811 22:23:06.961341 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:06.961389 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:06.962857 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:06.963432 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:06.963439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:06.963829 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:06.964413 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:06.964430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.965304 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.965308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:06.966385 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:06.966397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:06.966974 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:06.966979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:06.967151 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:06.967805 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:06.967823 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:06.969516 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:06.971373 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:06.971380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:06.971399 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:06.971715 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:06.971999 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:06.972002 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:06.972004 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:06.972188 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:06.972597 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:06.972604 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:06.972881 (+ 277us) client_negotiation.cc:769] Sending connection context
0811 22:23:06.973063 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 22:23:06.973431 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":142,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:23:06.975718 22458 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39005
I20260811 22:23:06.975958 22458 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:06.976500 22458 heartbeater.cc:511] Master 127.29.3.126:39005 requested a full tablet report, sending...
I20260811 22:23:06.977689 21973 ts_manager.cc:194] Registered new tserver with Master: f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
I20260811 22:23:06.978281 21973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:56243
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:23:06.997865 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39005
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:07.115464 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:07.115763 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:07.115809 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:07.119994 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:07.120072 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:07.120139 22473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:23:07.125758 22473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:07.126996 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:07.128116 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:07.138597 22480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:07.138597 22482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:07.138676 22486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:07.138938 22473 server_base.cc:1034] running on GCE node
I20260811 22:23:07.139452 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:07.140132 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:07.141335 22473 hybrid_clock.cc:648] HybridClock initialized: now 1786486987141313 us; error 36 us; skew 500 ppm
Aug 11 22:23:07 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486987, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:07.144302 22473 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:23:07.145287 22473 webserver.cc:460] Webserver started at http://127.29.3.67:39121/ using document root <none> and password file <none>
I20260811 22:23:07.145861 22473 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:07.145924 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:07.146111 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:07.148296 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "6e0326d3fa8a4381841179254b45cc6e"
format_stamp: "Formatted at 2026-08-11 22:23:07 on dist-test-slave-4tdj"
server_key: "447b4355dd489f746bc53b71d156454e"
server_key_iv: "05af7cd61ba12dd794ea30b93e192659"
server_key_version: "encryptionkey@0"
I20260811 22:23:07.148859 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "6e0326d3fa8a4381841179254b45cc6e"
format_stamp: "Formatted at 2026-08-11 22:23:07 on dist-test-slave-4tdj"
server_key: "447b4355dd489f746bc53b71d156454e"
server_key_iv: "05af7cd61ba12dd794ea30b93e192659"
server_key_version: "encryptionkey@0"
I20260811 22:23:07.152920 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:23:07.155653 22494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:07.157124 22473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:23:07.157269 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "6e0326d3fa8a4381841179254b45cc6e"
format_stamp: "Formatted at 2026-08-11 22:23:07 on dist-test-slave-4tdj"
server_key: "447b4355dd489f746bc53b71d156454e"
server_key_iv: "05af7cd61ba12dd794ea30b93e192659"
server_key_version: "encryptionkey@0"
I20260811 22:23:07.157407 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:07.173508 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:07.176872 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:07.177067 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:07.177704 22473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:07.178691 22473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:07.178757 22473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:07.178821 22473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:07.178848 22473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:07.188650 22473 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:46427
I20260811 22:23:07.188714 22673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:46427 every 8 connection(s)
I20260811 22:23:07.189632 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:23:07.195468 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 22473
I20260811 22:23:07.195585 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:23:07.195912 29709 external_mini_cluster.cc:1428] Setting key 6e51697ff762b55e41ef115bfb7c6f64
Aug 11 22:23:07 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486987, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:07.204155 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.191661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:38857 (local address 127.29.3.126:39005)
0811 22:23:07.191800 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.191805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.192737 (+ 932us) server_negotiation.cc:407] Connection header received
0811 22:23:07.194174 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.194179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.194245 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.194364 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:07.195643 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.196389 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.197703 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.197945 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.198053 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.200639 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:07.200643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:07.200645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:07.200675 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:07.202104 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:07.202674 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:07.202680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:07.202682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:07.202774 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:07.203081 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:07.203083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:07.203084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:07.203276 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:07.203353 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.203886 (+ 533us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.204018 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:23:07.205027 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.192036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:38857)
0811 22:23:07.192570 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.192619 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:07.193927 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:07.194552 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:07.194560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:07.194950 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:07.195469 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.195484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.196532 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.196535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.197579 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.197591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.198103 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.198108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.198350 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.198902 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:07.198920 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:07.200443 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:07.202213 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:07.202218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:07.202232 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:07.202553 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:07.202862 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:07.202865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:07.202866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:07.203011 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:07.203364 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:07.203370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:07.203639 (+ 269us) client_negotiation.cc:769] Sending connection context
0811 22:23:07.203811 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 22:23:07.204095 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":146,"threads_started":1}
I20260811 22:23:07.206334 22674 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39005
I20260811 22:23:07.206596 22674 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:07.207136 22674 heartbeater.cc:511] Master 127.29.3.126:39005 requested a full tablet report, sending...
I20260811 22:23:07.208302 21973 ts_manager.cc:194] Registered new tserver with Master: 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427)
I20260811 22:23:07.209012 21973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:38857
I20260811 22:23:07.209970 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:23:07.219313 22028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.212037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43926 (local address 127.29.3.126:39005)
0811 22:23:07.212189 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.212193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.212344 (+ 151us) server_negotiation.cc:407] Connection header received
0811 22:23:07.212460 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.212462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.212500 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.212571 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:07.213314 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.213827 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.214811 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.215031 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.215115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.216479 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:07.216482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:07.216484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:07.216508 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:07.217822 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:07.218235 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:07.218237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:07.218238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:07.218346 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:07.218556 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:07.218558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:07.218559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:07.218802 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:07.218908 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.219076 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.219180 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:23:07.222378 21973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:23:07.225499 21973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:23:07.241477 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.236044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.29.3.66:39279)
0811 22:23:07.237580 (+ 1536us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.237591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.237615 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:23:07.237688 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.237693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.237861 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.238002 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:07.238199 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.238986 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.240205 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.240771 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.240870 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.241021 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.241216 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.241312 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1416,"thread_start_us":138,"threads_started":1}
I20260811 22:23:07.241485 22701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.235358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39279 (local address 127.0.0.1:48226)
0811 22:23:07.236818 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.236832 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:07.236925 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:07.237989 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:07.237992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:07.238011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:07.238107 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.238113 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.239108 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.239112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.240101 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.240112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.240890 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.240894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.241017 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.241062 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 22:23:07.241171 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 22:23:07.241307 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1388,"thread_start_us":162,"threads_started":1}
I20260811 22:23:07.241523 22696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.235193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:46427 (local address 127.0.0.1:50838)
0811 22:23:07.236030 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.236094 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:07.236241 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:07.236978 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:07.236984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:07.237007 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:07.237159 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.237166 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.238413 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.238416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.239751 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.239761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.241085 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.241088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.241187 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.241210 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:23:07.241277 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 22:23:07.241350 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":86,"threads_started":1}
I20260811 22:23:07.241873 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.235403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.29.3.67:46427)
0811 22:23:07.235765 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.235777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.236380 (+ 603us) server_negotiation.cc:407] Connection header received
0811 22:23:07.236617 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.236624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.236816 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.236972 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:07.237289 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.238241 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.239887 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.240760 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.240852 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.241020 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.241606 (+ 586us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.241718 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":137,"threads_started":1}
I20260811 22:23:07.242247 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.236112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44655 (local address 127.0.0.1:40290)
0811 22:23:07.237220 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.237231 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:07.237322 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:07.237781 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:07.237784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:07.237804 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:07.237910 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.237916 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.239500 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.239503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.240594 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.240606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.241670 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.241673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.241966 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.241995 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 22:23:07.242049 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 22:23:07.242128 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":123,"threads_started":1}
I20260811 22:23:07.245302 22574 tablet_service.cc:1467] Processing CreateTablet for tablet ce6496e1fa674492a5a295f57222b517 (DEFAULT_TABLE table=test-table [id=15408749e7b64d6bae40bea22694ff28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:07.245302 22364 tablet_service.cc:1467] Processing CreateTablet for tablet ce6496e1fa674492a5a295f57222b517 (DEFAULT_TABLE table=test-table [id=15408749e7b64d6bae40bea22694ff28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:07.246237 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.236284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40290 (local address 127.29.3.65:44655)
0811 22:23:07.236590 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.236597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.237324 (+ 727us) server_negotiation.cc:407] Connection header received
0811 22:23:07.237437 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.237442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.237638 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.237790 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:07.238045 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.239215 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.240759 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.241540 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.242541 (+ 1001us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.242692 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.242742 (+ 50us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.242833 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260811 22:23:07.246645 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6496e1fa674492a5a295f57222b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:07.246655 22574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6496e1fa674492a5a295f57222b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:07.248978 22154 tablet_service.cc:1467] Processing CreateTablet for tablet ce6496e1fa674492a5a295f57222b517 (DEFAULT_TABLE table=test-table [id=15408749e7b64d6bae40bea22694ff28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:07.250095 22154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6496e1fa674492a5a295f57222b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:07.254729 22704 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Bootstrap starting.
I20260811 22:23:07.255221 22705 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Bootstrap starting.
I20260811 22:23:07.257874 22706 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Bootstrap starting.
I20260811 22:23:07.258266 22705 tablet_bootstrap.cc:654] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:07.259068 22705 log.cc:824] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:07.260042 22706 tablet_bootstrap.cc:654] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:07.260825 22706 log.cc:824] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:07.261389 22705 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: No bootstrap required, opened a new log
I20260811 22:23:07.261603 22705 ts_tablet_manager.cc:1397] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260811 22:23:07.261670 22704 tablet_bootstrap.cc:654] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:07.262782 22704 log.cc:824] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:07.263082 22706 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: No bootstrap required, opened a new log
I20260811 22:23:07.263334 22706 ts_tablet_manager.cc:1397] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:23:07.264984 22704 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: No bootstrap required, opened a new log
I20260811 22:23:07.265213 22704 ts_tablet_manager.cc:1397] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260811 22:23:07.265172 22705 raft_consensus.cc:348] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.265411 22705 raft_consensus.cc:374] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:07.265445 22705 raft_consensus.cc:729] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f358c50f28684a878bd82a66f0498a39, State: Initialized, Role: FOLLOWER
I20260811 22:23:07.265887 22705 consensus_queue.cc:260] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.266620 22458 heartbeater.cc:503] Master 127.29.3.126:39005 was elected leader, sending a full tablet report...
I20260811 22:23:07.268041 22705 ts_tablet_manager.cc:1428] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260811 22:23:07.268025 22706 raft_consensus.cc:348] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.268425 22706 raft_consensus.cc:374] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:07.268524 22706 raft_consensus.cc:729] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861058085af44412b85f3256353ea8b4, State: Initialized, Role: FOLLOWER
I20260811 22:23:07.269086 22706 consensus_queue.cc:260] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.269183 22704 raft_consensus.cc:348] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.269461 22704 raft_consensus.cc:374] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:07.269550 22704 raft_consensus.cc:729] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0326d3fa8a4381841179254b45cc6e, State: Initialized, Role: FOLLOWER
I20260811 22:23:07.269976 22253 heartbeater.cc:503] Master 127.29.3.126:39005 was elected leader, sending a full tablet report...
I20260811 22:23:07.270423 22706 ts_tablet_manager.cc:1428] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 22:23:07.270428 22704 consensus_queue.cc:260] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.271409 22704 ts_tablet_manager.cc:1428] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:23:07.271919 22674 heartbeater.cc:503] Master 127.29.3.126:39005 was elected leader, sending a full tablet report...
W20260811 22:23:07.440711 22677 tablet.cc:2369] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:23:07.459820 22459 tablet.cc:2369] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:23:07.464799 22254 tablet.cc:2369] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:23:07.500372 22714 raft_consensus.cc:482] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:07.500629 22714 raft_consensus.cc:504] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.501881 22714 leader_election.cc:288] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279), 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655)
I20260811 22:23:07.505856 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.502313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:45917 (local address 127.29.3.66:39279)
0811 22:23:07.502444 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.502448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.502803 (+ 355us) server_negotiation.cc:407] Connection header received
0811 22:23:07.502929 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.502932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.502987 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.503071 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:07.503338 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.503889 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.504928 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.505332 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.505432 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.505493 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.505663 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.505731 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:23:07.505859 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.502335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:50535 (local address 127.29.3.65:44655)
0811 22:23:07.502446 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.502450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.502466 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:23:07.502529 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.502532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.502587 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.502727 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:07.503023 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.503639 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.504771 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.505261 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.505344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.505407 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.505629 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.505726 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:23:07.505856 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.502170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44655 (local address 127.29.3.67:50535)
0811 22:23:07.502318 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.502334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:07.502439 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:07.502734 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:07.502737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:07.502756 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:07.502909 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.502916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.503755 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.503759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.504638 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.504649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.505377 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.505381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.505486 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.505508 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:23:07.505602 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 22:23:07.505700 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":7,"spinlock_wait_cycles":31104}
I20260811 22:23:07.505885 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.502170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39279 (local address 127.29.3.67:45917)
0811 22:23:07.502691 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.502706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:07.502827 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:07.503078 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:07.503081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:07.503098 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:07.503233 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.503240 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.503994 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.503996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.504804 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:07.504815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.505455 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.505459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:07.505538 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.505559 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:23:07.505634 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 22:23:07.505738 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":176,"spinlock_wait_cycles":896,"thread_start_us":125,"threads_started":1}
I20260811 22:23:07.506680 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6496e1fa674492a5a295f57222b517" candidate_uuid: "6e0326d3fa8a4381841179254b45cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f358c50f28684a878bd82a66f0498a39" is_pre_election: true
I20260811 22:23:07.506655 22187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6496e1fa674492a5a295f57222b517" candidate_uuid: "6e0326d3fa8a4381841179254b45cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861058085af44412b85f3256353ea8b4" is_pre_election: true
I20260811 22:23:07.507014 22187 raft_consensus.cc:2454] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e0326d3fa8a4381841179254b45cc6e in term 0.
I20260811 22:23:07.507014 22396 raft_consensus.cc:2454] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e0326d3fa8a4381841179254b45cc6e in term 0.
I20260811 22:23:07.507517 22500 leader_election.cc:302] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39; no voters:
I20260811 22:23:07.507824 22714 raft_consensus.cc:2781] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:23:07.507906 22714 raft_consensus.cc:482] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:23:07.507938 22714 raft_consensus.cc:3037] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:07.508991 22714 raft_consensus.cc:504] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.509369 22714 leader_election.cc:288] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [CANDIDATE]: Term 1 election: Requested vote from peers f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279), 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655)
I20260811 22:23:07.509825 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6496e1fa674492a5a295f57222b517" candidate_uuid: "6e0326d3fa8a4381841179254b45cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f358c50f28684a878bd82a66f0498a39"
I20260811 22:23:07.509825 22187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6496e1fa674492a5a295f57222b517" candidate_uuid: "6e0326d3fa8a4381841179254b45cc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861058085af44412b85f3256353ea8b4"
I20260811 22:23:07.510020 22396 raft_consensus.cc:3037] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:07.510026 22187 raft_consensus.cc:3037] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:07.511420 22187 raft_consensus.cc:2454] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e0326d3fa8a4381841179254b45cc6e in term 1.
I20260811 22:23:07.511420 22396 raft_consensus.cc:2454] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e0326d3fa8a4381841179254b45cc6e in term 1.
I20260811 22:23:07.511720 22500 leader_election.cc:302] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39; no voters:
I20260811 22:23:07.511891 22714 raft_consensus.cc:2781] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:07.512161 22714 raft_consensus.cc:686] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 1 LEADER]: Becoming Leader. State: Replica: 6e0326d3fa8a4381841179254b45cc6e, State: Running, Role: LEADER
I20260811 22:23:07.512527 22714 consensus_queue.cc:237] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:07.516160 21973 catalog_manager.cc:5495] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67). New cstate: current_term: 1 leader_uuid: "6e0326d3fa8a4381841179254b45cc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } health_report { overall_health: UNKNOWN } } }
I20260811 22:23:07.541594 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.538211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.29.3.67:46427)
0811 22:23:07.538375 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:07.538379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:07.538412 (+ 33us) server_negotiation.cc:407] Connection header received
0811 22:23:07.538492 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:07.538495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:07.538554 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:07.538636 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:23:07.538836 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.539341 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.540117 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:07.540334 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:07.540409 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:07.540705 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:23:07.541172 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:23:07.541268 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:07.541380 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 22:23:07.541440 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:23:07.548173 22187 raft_consensus.cc:1261] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Refusing update from remote peer 6e0326d3fa8a4381841179254b45cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:07.548173 22396 raft_consensus.cc:1261] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Refusing update from remote peer 6e0326d3fa8a4381841179254b45cc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:07.549032 22721 consensus_queue.cc:1035] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:07.549247 22714 consensus_queue.cc:1035] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:07.556938 22726 mvcc.cc:205] Tried to move back new op lower bound from 7317450700988792832 to 7317450700854235136. Current Snapshot: MvccSnapshot[applied={T|T < 7317450700988792832}]
I20260811 22:23:07.559285 22729 mvcc.cc:205] Tried to move back new op lower bound from 7317450700988792832 to 7317450700854235136. Current Snapshot: MvccSnapshot[applied={T|T < 7317450700988792832}]
I20260811 22:23:07.567059 21972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:07.567243 21972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:07.569553 21972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Sending DeleteTablet for 3 replicas of tablet ce6496e1fa674492a5a295f57222b517
I20260811 22:23:07.570403 22364 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6496e1fa674492a5a295f57222b517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:07 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:48226
I20260811 22:23:07.570403 22574 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6496e1fa674492a5a295f57222b517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:07 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:50838
I20260811 22:23:07.570484 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 22064
I20260811 22:23:07.570533 22154 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6496e1fa674492a5a295f57222b517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:07 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:40290
I20260811 22:23:07.583900 22736 tablet_replica.cc:330] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: stopping tablet replica
I20260811 22:23:07.584477 22736 raft_consensus.cc:2229] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:07.584858 22738 tablet_replica.cc:330] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: stopping tablet replica
I20260811 22:23:07.585422 22736 raft_consensus.cc:2258] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:07.585843 22738 raft_consensus.cc:2229] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:23:07.586649 22738 raft_consensus.cc:2258] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:07.587980 22736 ts_tablet_manager.cc:1905] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:07.588510 22738 ts_tablet_manager.cc:1905] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:07.591181 22736 ts_tablet_manager.cc:1918] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:07.591394 22736 log.cc:1196] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/ce6496e1fa674492a5a295f57222b517
I20260811 22:23:07.591389 22738 ts_tablet_manager.cc:1918] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:07.591480 22738 log.cc:1196] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/ce6496e1fa674492a5a295f57222b517
I20260811 22:23:07.591809 22736 ts_tablet_manager.cc:1939] T ce6496e1fa674492a5a295f57222b517 P f358c50f28684a878bd82a66f0498a39: Deleting consensus metadata
I20260811 22:23:07.591964 22738 ts_tablet_manager.cc:1939] T ce6496e1fa674492a5a295f57222b517 P 6e0326d3fa8a4381841179254b45cc6e: Deleting consensus metadata
I20260811 22:23:07.593294 21956 catalog_manager.cc:4841] TS f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): tablet ce6496e1fa674492a5a295f57222b517 (table test-table [id=15408749e7b64d6bae40bea22694ff28]) successfully deleted
I20260811 22:23:07.593351 21957 catalog_manager.cc:4841] TS 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): tablet ce6496e1fa674492a5a295f57222b517 (table test-table [id=15408749e7b64d6bae40bea22694ff28]) successfully deleted
W20260811 22:23:07.594045 21958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:44655 (error 108)
I20260811 22:23:07.595124 22697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:07.594798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44655 (local address 127.0.0.1:40298)
0811 22:23:07.594938 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:07.595018 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260811 22:23:07.595219 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 22260
W20260811 22:23:07.613237 21958 catalog_manager.cc:4584] TS 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce6496e1fa674492a5a295f57222b517: Network error: Client connection negotiation failed: client connection to 127.29.3.65:44655: connect: Connection refused (error 111)
I20260811 22:23:07.628533 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 22473
I20260811 22:23:07.647967 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 21926
I20260811 22:23:07.675580 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39005
--webserver_interface=127.29.3.126
--webserver_port=39965
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:07.794071 22742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:07.794456 22742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:07.794536 22742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:07.798800 22742 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:07.798883 22742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:07.798913 22742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:07.798934 22742 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:07.798954 22742 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:07.804595 22742 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=39965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:07.805848 22742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:07.806985 22742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:07.821627 22751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:07.821627 22755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:07.821791 22742 server_base.cc:1034] running on GCE node
W20260811 22:23:07.821964 22758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:07.822674 22742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:07.823926 22742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:07.825090 22742 hybrid_clock.cc:648] HybridClock initialized: now 1786486987825065 us; error 41 us; skew 500 ppm
Aug 11 22:23:07 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486987, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:07.828009 22742 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:07.829058 22742 webserver.cc:460] Webserver started at http://127.29.3.126:39965/ using document root <none> and password file <none>
I20260811 22:23:07.829674 22742 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:07.829758 22742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:07.833932 22742 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:07.836375 22769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:07.837738 22742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:23:07.837872 22742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "cce39c1cd1104ed3a181790c0eae3276"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "2b0a93c501096d2982e8a5c77b7da961"
server_key_iv: "97dedf1f312e9d8b959ebf24e58f2bb2"
server_key_version: "encryptionkey@0"
I20260811 22:23:07.838619 22742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:07.861429 22742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:07.864949 22742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:07.865166 22742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:07.873165 22742 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:39005
I20260811 22:23:07.873481 22849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:39005 every 8 connection(s)
I20260811 22:23:07.874452 22742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:07.879504 22850 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:23:07.881424 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap starting.
I20260811 22:23:07.883667 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 22742
I20260811 22:23:07.884181 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:44655
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:39005
--webserver_port=33841
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:07.900830 22850 log.cc:824] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:07.910706 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:23:07.911178 22850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap complete.
I20260811 22:23:07.915532 22850 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:07.915966 22850 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce39c1cd1104ed3a181790c0eae3276, State: Initialized, Role: FOLLOWER
I20260811 22:23:07.916495 22850 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:07.916632 22850 raft_consensus.cc:388] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:23:07.916712 22850 raft_consensus.cc:482] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:23:07.916812 22850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:23:07.918995 22850 raft_consensus.cc:504] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:07.919384 22850 leader_election.cc:302] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cce39c1cd1104ed3a181790c0eae3276; no voters:
I20260811 22:23:07.919761 22850 leader_election.cc:288] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:23:07.919870 22858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:23:07.920089 22858 raft_consensus.cc:686] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 LEADER]: Becoming Leader. State: Replica: cce39c1cd1104ed3a181790c0eae3276, State: Running, Role: LEADER
I20260811 22:23:07.920475 22858 consensus_queue.cc:237] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:07.921155 22850 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:07.922158 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cce39c1cd1104ed3a181790c0eae3276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } } }
I20260811 22:23:07.922210 22864 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cce39c1cd1104ed3a181790c0eae3276. Latest consensus state: current_term: 2 leader_uuid: "cce39c1cd1104ed3a181790c0eae3276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } } }
I20260811 22:23:07.922314 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:07.922372 22864 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:07.926889 22891 catalog_manager.cc:1259] Loaded cluster ID: 6733f95e016d4ccc8ca7c87201751c46
I20260811 22:23:07.926962 22891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: loading cluster ID for follower catalog manager: success
W20260811 22:23:07.927794 22891 catalog_manager.cc:1576] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260811 22:23:07.927863 22891 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260811 22:23:07.927937 22877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:07.930526 22877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:07.930949 22877 catalog_manager.cc:1259] Loaded cluster ID: 6733f95e016d4ccc8ca7c87201751c46
I20260811 22:23:07.931002 22877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:23:07.931838 22877 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 22:23:07.934770 22877 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 22:23:07.939857 22877 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:23:07.940615 22877 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Loaded TSK: 0
I20260811 22:23:07.941710 22877 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:23:08.022501 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:08.022856 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:08.022972 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:08.027974 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:08.028064 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:08.028152 22853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:23:08.035410 22853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:44655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=33841
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:08.036639 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:08.037814 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:08.048975 22906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.048974 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.049103 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:08.049386 22853 server_base.cc:1034] running on GCE node
I20260811 22:23:08.049984 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:08.050976 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:08.052207 22853 hybrid_clock.cc:648] HybridClock initialized: now 1786486988052169 us; error 49 us; skew 500 ppm
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:08.056118 22853 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:23:08.057510 22853 webserver.cc:460] Webserver started at http://127.29.3.65:33841/ using document root <none> and password file <none>
I20260811 22:23:08.058399 22853 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:08.058526 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:08.064267 22853 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:23:08.067335 22918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.069067 22853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:08.069223 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "861058085af44412b85f3256353ea8b4"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "f97c18368d8681e647f760a16f7a802d"
server_key_iv: "819d087d4dc4869d819c307693fa6906"
server_key_version: "encryptionkey@0"
I20260811 22:23:08.069964 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:08.102988 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:08.106146 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:08.106359 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:08.107020 22853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:08.108201 22928 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 22:23:08.112818 22853 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 22:23:08.112915 22853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260811 22:23:08.112995 22853 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 22:23:08.114991 22853 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 22:23:08.115051 22853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.001s
I20260811 22:23:08.115366 22928 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Bootstrap starting.
I20260811 22:23:08.125469 22853 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:44655
I20260811 22:23:08.125556 23105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:44655 every 8 connection(s)
I20260811 22:23:08.126580 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:23:08.133296 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 22853
I20260811 22:23:08.133813 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:39279
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:39005
--webserver_port=37117
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:08.137234 22928 log.cc:824] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:08.146232 22928 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:23:08.146837 22928 tablet_bootstrap.cc:492] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Bootstrap complete.
I20260811 22:23:08.147010 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.128695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:47109 (local address 127.29.3.126:39005)
0811 22:23:08.130564 (+ 1869us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.130580 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.130621 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:23:08.130888 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.130899 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.131238 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.131520 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:08.133246 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.134777 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.136470 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.137155 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.137305 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.140408 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:08.140415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:08.140433 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:08.140471 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:08.142696 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.143582 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.143588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.143593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.143769 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.144192 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.144195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.144197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.144702 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:08.144838 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.145605 (+ 767us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.145928 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1727,"thread_start_us":120,"threads_started":1}
I20260811 22:23:08.147027 23109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.128568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:47109)
0811 22:23:08.129161 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.129233 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.130392 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.131699 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.131709 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.132260 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:08.132968 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.133029 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.134964 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.134968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.136313 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.136327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.137325 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.137329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.137517 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.138148 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:08.138174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:08.140144 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:08.142857 (+ 2713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.142864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.142880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.143433 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.143879 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.143883 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.143885 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.144077 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.144836 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:08.144843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:08.145207 (+ 364us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.145458 (+ 251us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.145860 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":158,"threads_started":1}
I20260811 22:23:08.148258 22928 ts_tablet_manager.cc:1397] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Time spent bootstrapping tablet: real 0.033s user 0.027s sys 0.005s
I20260811 22:23:08.150125 23106 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39005
I20260811 22:23:08.150450 23106 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:08.151101 23106 heartbeater.cc:511] Master 127.29.3.126:39005 requested a full tablet report, sending...
I20260811 22:23:08.151348 22928 raft_consensus.cc:348] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.151839 22928 raft_consensus.cc:729] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861058085af44412b85f3256353ea8b4, State: Initialized, Role: FOLLOWER
I20260811 22:23:08.152366 22928 consensus_queue.cc:260] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.154842 22928 ts_tablet_manager.cc:1428] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260811 22:23:08.155615 22794 ts_manager.cc:194] Registered new tserver with Master: 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655)
I20260811 22:23:08.159257 22794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:47109
I20260811 22:23:08.162063 23106 heartbeater.cc:503] Master 127.29.3.126:39005 was elected leader, sending a full tablet report...
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486987, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:23:08.169773 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.158663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44655 (local address 127.0.0.1:32978)
0811 22:23:08.159502 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.159554 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.159832 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.160533 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.160537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.160886 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:08.160995 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.161002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.161974 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.161977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.162784 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.162795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.163526 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.163530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.163642 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.164210 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:08.164222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:08.165868 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:08.167974 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.167984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.167989 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.168229 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.168542 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.168552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.168554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.168640 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.169191 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:08.169196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:08.169315 (+ 119us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.169433 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.169605 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":138,"threads_started":1}
I20260811 22:23:08.169925 23116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.158777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.29.3.65:44655)
0811 22:23:08.159215 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.159228 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.159684 (+ 456us) server_negotiation.cc:407] Connection header received
0811 22:23:08.160015 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.160021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.160236 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.160380 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:08.161128 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.161846 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.162891 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.163390 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.163509 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.166031 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:08.166043 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:08.166058 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:08.166089 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:08.167848 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.168339 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.168346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.168348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.168447 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.168732 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.168735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.168737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.169063 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:08.169195 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.169532 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.169728 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":105,"threads_started":1}
I20260811 22:23:08.171025 22995 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6496e1fa674492a5a295f57222b517 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:07 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:32978
I20260811 22:23:08.171906 23118 tablet_replica.cc:330] stopping tablet replica
I20260811 22:23:08.172219 23118 raft_consensus.cc:2229] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:08.172469 23118 raft_consensus.cc:2258] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:08.173470 23118 ts_tablet_manager.cc:1905] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:08.177495 23118 ts_tablet_manager.cc:1918] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:08.177590 23118 log.cc:1196] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/ce6496e1fa674492a5a295f57222b517
I20260811 22:23:08.177824 23118 ts_tablet_manager.cc:1939] T ce6496e1fa674492a5a295f57222b517 P 861058085af44412b85f3256353ea8b4: Deleting consensus metadata
I20260811 22:23:08.179724 22774 catalog_manager.cc:4841] TS 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655): tablet ce6496e1fa674492a5a295f57222b517 (table test-table [id=15408749e7b64d6bae40bea22694ff28]) successfully deleted
W20260811 22:23:08.261548 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:08.261833 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:08.261879 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:08.265950 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:08.266029 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:08.266088 23111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:23:08.271623 23111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:39279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=37117
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:08.272758 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:08.273801 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:08.284107 23131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.284111 23129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.284125 23128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:08.284410 23111 server_base.cc:1034] running on GCE node
I20260811 22:23:08.284847 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:08.285550 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:08.286705 23111 hybrid_clock.cc:648] HybridClock initialized: now 1786486988286675 us; error 43 us; skew 500 ppm
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:08.289466 23111 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:23:08.290496 23111 webserver.cc:460] Webserver started at http://127.29.3.66:37117/ using document root <none> and password file <none>
I20260811 22:23:08.291071 23111 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:08.291152 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:08.295253 23111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:08.297549 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.298925 23111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:23:08.299054 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "f358c50f28684a878bd82a66f0498a39"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "dda47d301bad079ad70db636d6a0698d"
server_key_iv: "c18e8a0765cce04b8e8e2b60c9337201"
server_key_version: "encryptionkey@0"
I20260811 22:23:08.299687 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:08.316931 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:08.320353 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:08.320555 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:08.321267 23111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:08.322319 23111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:08.322388 23111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.322449 23111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:08.322477 23111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.332134 23111 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:39279
I20260811 22:23:08.332165 23334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:39279 every 8 connection(s)
I20260811 22:23:08.333213 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:23:08.342900 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 23111
I20260811 22:23:08.343348 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:46427
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:39005
--webserver_port=39121
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:08.347884 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.335191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:43839 (local address 127.29.3.126:39005)
0811 22:23:08.335333 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.335337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.336258 (+ 921us) server_negotiation.cc:407] Connection header received
0811 22:23:08.337674 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.337678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.337748 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.337827 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:08.339250 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.339863 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.341044 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.341280 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.341374 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.344130 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:08.344133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:08.344136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:08.344166 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:08.345547 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.346207 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.346210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.346212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.346330 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.346708 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.346710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.346711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.346896 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:08.346971 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.347616 (+ 645us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.347750 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:23:08.349151 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.335604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:43839)
0811 22:23:08.336101 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.336147 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.337445 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.338039 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.338047 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.338543 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:08.339093 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.339107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.339995 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.339998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.340917 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.340927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.341435 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.341439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.341611 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.342230 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:08.342248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:08.343940 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:08.345675 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.345683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.345704 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.346067 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.346419 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.346422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.346424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.346623 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.346994 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:08.347001 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:08.347371 (+ 370us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.347595 (+ 224us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.347977 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":122,"threads_started":1}
I20260811 22:23:08.350582 23335 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39005
I20260811 22:23:08.350888 23335 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:08.351593 23335 heartbeater.cc:511] Master 127.29.3.126:39005 requested a full tablet report, sending...
I20260811 22:23:08.352972 22794 ts_manager.cc:194] Registered new tserver with Master: f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
I20260811 22:23:08.353857 22794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:43839
W20260811 22:23:08.465121 23340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:08.465447 23340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:08.465494 23340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:08.469646 23340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:08.469730 23340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:08.469792 23340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:23:08.475415 23340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:46427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=39121
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:08.476572 23340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:08.477593 23340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:08.487567 23348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.487555 23351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.487555 23349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:08.487780 23340 server_base.cc:1034] running on GCE node
I20260811 22:23:08.488322 23340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:08.489022 23340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:08.490200 23340 hybrid_clock.cc:648] HybridClock initialized: now 1786486988490181 us; error 39 us; skew 500 ppm
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:08.493026 23340 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:23:08.493969 23340 webserver.cc:460] Webserver started at http://127.29.3.67:39121/ using document root <none> and password file <none>
I20260811 22:23:08.494589 23340 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:08.494679 23340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:08.498710 23340 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:08.501024 23363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.502562 23340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:08.502701 23340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "6e0326d3fa8a4381841179254b45cc6e"
format_stamp: "Formatted at 2026-08-11 22:23:07 on dist-test-slave-4tdj"
server_key: "447b4355dd489f746bc53b71d156454e"
server_key_iv: "05af7cd61ba12dd794ea30b93e192659"
server_key_version: "encryptionkey@0"
I20260811 22:23:08.503356 23340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:08.518090 23340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:08.521139 23340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:08.521373 23340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:08.522035 23340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:08.523064 23340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:08.523114 23340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.523172 23340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:08.523201 23340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.532239 23340 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:46427
I20260811 22:23:08.532269 23565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:46427 every 8 connection(s)
I20260811 22:23:08.533283 23340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:23:08.542784 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 23340
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:08.547715 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.535240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:46713 (local address 127.29.3.126:39005)
0811 22:23:08.535421 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.535426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.536353 (+ 927us) server_negotiation.cc:407] Connection header received
0811 22:23:08.537744 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.537747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.537801 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.537876 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:08.539170 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.539718 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.541090 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.541336 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.541421 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.544225 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:08.544229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:08.544231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:08.544258 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:08.545611 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.546145 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.546148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.546150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.546233 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.546587 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.546591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.546592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.546774 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:08.546860 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.547402 (+ 542us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.547539 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:23:08.548586 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.535677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:46713)
0811 22:23:08.536184 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.536232 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.537532 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.538044 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.538051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.538472 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:08.539024 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.539039 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.539874 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.539877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.540951 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.540963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.541496 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.541500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.541663 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.542326 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:08.542349 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:08.544056 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:08.545740 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.545747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.545765 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.546049 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.546345 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:08.546348 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:08.546350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:08.546494 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:08.546876 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:08.546884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:08.547154 (+ 270us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.547325 (+ 171us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.547686 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":151,"threads_started":1}
I20260811 22:23:08.549844 23566 heartbeater.cc:347] Connected to a master server at 127.29.3.126:39005
I20260811 22:23:08.550120 23566 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:08.550757 23566 heartbeater.cc:511] Master 127.29.3.126:39005 requested a full tablet report, sending...
I20260811 22:23:08.551541 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.543476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.29.3.126:39005)
0811 22:23:08.543776 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.543781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.543863 (+ 82us) server_negotiation.cc:407] Connection header received
0811 22:23:08.543991 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.543994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.544042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.544114 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:08.544921 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.545549 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.546398 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.546633 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.546721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.547980 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:08.547983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:08.547986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:08.548010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:08.549507 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.549930 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.549934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.549935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.550029 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:08.550352 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:08.550356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:08.550358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:08.550552 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:08.550747 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.551287 (+ 540us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.551413 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"spinlock_wait_cycles":768,"thread_start_us":80,"threads_started":1}
I20260811 22:23:08.551959 22794 ts_manager.cc:194] Registered new tserver with Master: 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427)
I20260811 22:23:08.552613 22794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:46713
I20260811 22:23:08.553475 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:23:08.557760 23573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.554080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.29.3.126:39005)
0811 22:23:08.554188 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.554192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.554462 (+ 270us) server_negotiation.cc:407] Connection header received
0811 22:23:08.554606 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.554609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.554709 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.554796 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:23:08.555025 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.555537 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.556376 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.556599 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.556700 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.556990 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:23:08.557379 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:23:08.557485 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.557557 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.557618 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:23:08.558491 22792 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:23:08.561826 22792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:23:08.570706 22995 tablet_service.cc:1467] Processing CreateTablet for tablet d826229fbcc9485fa366539c1e474fd9 (DEFAULT_TABLE table=test-table [id=29092a6db86b40bf97a1d18ec6652174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:08.571421 22995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d826229fbcc9485fa366539c1e474fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:08.574573 22928 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Bootstrap starting.
I20260811 22:23:08.575399 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.569349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:46427 (local address 127.0.0.1:33736)
0811 22:23:08.569887 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.569904 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.570004 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.571089 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.571092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.571113 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:08.571229 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.571235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.572701 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.572705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.573641 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.573652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.574900 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.574904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.575007 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.575064 (+ 57us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.575148 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.575237 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460}
I20260811 22:23:08.575513 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.569703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.29.3.67:46427)
0811 22:23:08.570511 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.570522 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.570540 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:23:08.570601 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.570605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.570822 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.570995 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:08.571359 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.572550 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.573775 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.574539 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.574662 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.574833 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.575241 (+ 408us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.575358 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":154,"threads_started":1}
I20260811 22:23:08.575543 22928 tablet_bootstrap.cc:654] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:08.575737 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.569576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39279 (local address 127.0.0.1:44412)
0811 22:23:08.570110 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.570123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.570179 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.571005 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.571008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.571027 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:08.571142 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.571149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.573022 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.573025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.574148 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.574161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.575348 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.575351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.575445 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.575470 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.575521 (+ 51us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.575598 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":130,"threads_started":1}
I20260811 22:23:08.576313 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.569727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.29.3.66:39279)
0811 22:23:08.570434 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.570448 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.570475 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:23:08.570563 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.570569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.570823 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.571284 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:08.571463 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.572883 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.574330 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.575220 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.575917 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.576050 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.576105 (+ 55us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.576182 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":122,"threads_started":1}
I20260811 22:23:08.577050 22928 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: No bootstrap required, opened a new log
I20260811 22:23:08.577150 22928 ts_tablet_manager.cc:1397] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 22:23:08.577636 22928 raft_consensus.cc:348] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.577777 22928 raft_consensus.cc:374] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:08.577817 22928 raft_consensus.cc:729] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861058085af44412b85f3256353ea8b4, State: Initialized, Role: FOLLOWER
I20260811 22:23:08.577975 22928 consensus_queue.cc:260] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.578361 22928 ts_tablet_manager.cc:1428] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260811 22:23:08.578938 23444 tablet_service.cc:1467] Processing CreateTablet for tablet d826229fbcc9485fa366539c1e474fd9 (DEFAULT_TABLE table=test-table [id=29092a6db86b40bf97a1d18ec6652174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:08.579217 23226 tablet_service.cc:1467] Processing CreateTablet for tablet d826229fbcc9485fa366539c1e474fd9 (DEFAULT_TABLE table=test-table [id=29092a6db86b40bf97a1d18ec6652174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:08.580061 23444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d826229fbcc9485fa366539c1e474fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:08.580353 23226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d826229fbcc9485fa366539c1e474fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:08.587507 23587 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Bootstrap starting.
I20260811 22:23:08.588310 23588 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Bootstrap starting.
I20260811 22:23:08.590583 23588 tablet_bootstrap.cc:654] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:08.590583 23587 tablet_bootstrap.cc:654] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:08.591576 23587 log.cc:824] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:08.591853 23588 log.cc:824] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:08.593606 23588 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: No bootstrap required, opened a new log
I20260811 22:23:08.593601 23587 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: No bootstrap required, opened a new log
I20260811 22:23:08.593806 23588 ts_tablet_manager.cc:1397] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:23:08.593830 23587 ts_tablet_manager.cc:1397] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260811 22:23:08.596932 23588 raft_consensus.cc:348] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.596933 23587 raft_consensus.cc:348] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.597175 23587 raft_consensus.cc:374] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:08.597178 23588 raft_consensus.cc:374] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:08.597208 23587 raft_consensus.cc:729] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f358c50f28684a878bd82a66f0498a39, State: Initialized, Role: FOLLOWER
I20260811 22:23:08.597229 23588 raft_consensus.cc:729] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0326d3fa8a4381841179254b45cc6e, State: Initialized, Role: FOLLOWER
I20260811 22:23:08.597635 23587 consensus_queue.cc:260] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.597635 23588 consensus_queue.cc:260] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.598402 23335 heartbeater.cc:503] Master 127.29.3.126:39005 was elected leader, sending a full tablet report...
I20260811 22:23:08.598629 23566 heartbeater.cc:503] Master 127.29.3.126:39005 was elected leader, sending a full tablet report...
I20260811 22:23:08.598670 23587 ts_tablet_manager.cc:1428] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:23:08.599632 23588 ts_tablet_manager.cc:1428] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260811 22:23:08.627921 23107 tablet.cc:2369] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:23:08.680240 23591 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:08.680550 23591 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.681763 23591 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655)
I20260811 22:23:08.685861 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.682015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:46427 (local address 127.29.3.66:45025)
0811 22:23:08.682160 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.682176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.682268 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.682591 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.682593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.682615 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:08.682724 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.682729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.683576 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.683579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.684587 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.684600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.685489 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.685492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.685584 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.685608 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.685678 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.685740 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260811 22:23:08.685915 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.682200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:45025 (local address 127.29.3.67:46427)
0811 22:23:08.682364 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.682369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.682382 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:23:08.682432 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.682434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.682477 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.682555 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:08.682843 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.683478 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.684734 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.685351 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.685451 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.685521 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.685709 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.685786 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:23:08.686757 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.682143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:57507 (local address 127.29.3.65:44655)
0811 22:23:08.682423 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.682428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.682778 (+ 350us) server_negotiation.cc:407] Connection header received
0811 22:23:08.682867 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.682869 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.682923 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.683014 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:08.683281 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.683934 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.685585 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.686209 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.686327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.686519 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.686556 (+ 37us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.686629 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260811 22:23:08.686765 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.682034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44655 (local address 127.29.3.66:57507)
0811 22:23:08.682687 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.682700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:08.682782 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:08.683041 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:08.683044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:08.683062 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:08.683182 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.683188 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.684076 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.684079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.685335 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:08.685347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.686354 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.686358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:08.686453 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.686473 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 22:23:08.686534 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 22:23:08.686614 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":356,"thread_start_us":109,"threads_started":1}
I20260811 22:23:08.686721 23486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d826229fbcc9485fa366539c1e474fd9" candidate_uuid: "f358c50f28684a878bd82a66f0498a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0326d3fa8a4381841179254b45cc6e" is_pre_election: true
I20260811 22:23:08.687078 23486 raft_consensus.cc:2454] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f358c50f28684a878bd82a66f0498a39 in term 0.
I20260811 22:23:08.687634 23146 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39; no voters:
I20260811 22:23:08.687578 23032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d826229fbcc9485fa366539c1e474fd9" candidate_uuid: "f358c50f28684a878bd82a66f0498a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861058085af44412b85f3256353ea8b4" is_pre_election: true
I20260811 22:23:08.687897 23032 raft_consensus.cc:2454] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f358c50f28684a878bd82a66f0498a39 in term 0.
I20260811 22:23:08.687911 23591 raft_consensus.cc:2781] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:23:08.688007 23591 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:23:08.688061 23591 raft_consensus.cc:3037] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:08.689258 23591 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.689603 23591 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [CANDIDATE]: Term 1 election: Requested vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655)
I20260811 22:23:08.690003 23486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d826229fbcc9485fa366539c1e474fd9" candidate_uuid: "f358c50f28684a878bd82a66f0498a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0326d3fa8a4381841179254b45cc6e"
I20260811 22:23:08.690003 23032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d826229fbcc9485fa366539c1e474fd9" candidate_uuid: "f358c50f28684a878bd82a66f0498a39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861058085af44412b85f3256353ea8b4"
I20260811 22:23:08.690153 23032 raft_consensus.cc:3037] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:08.690153 23486 raft_consensus.cc:3037] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:08.691555 23486 raft_consensus.cc:2454] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f358c50f28684a878bd82a66f0498a39 in term 1.
I20260811 22:23:08.691553 23032 raft_consensus.cc:2454] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f358c50f28684a878bd82a66f0498a39 in term 1.
I20260811 22:23:08.691941 23147 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 861058085af44412b85f3256353ea8b4, f358c50f28684a878bd82a66f0498a39; no voters:
I20260811 22:23:08.692159 23591 raft_consensus.cc:2781] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:08.692427 23591 raft_consensus.cc:686] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 1 LEADER]: Becoming Leader. State: Replica: f358c50f28684a878bd82a66f0498a39, State: Running, Role: LEADER
I20260811 22:23:08.692775 23591 consensus_queue.cc:237] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:08.696390 22794 catalog_manager.cc:5495] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 reported cstate change: term changed from 0 to 1, leader changed from <none> to f358c50f28684a878bd82a66f0498a39 (127.29.3.66). New cstate: current_term: 1 leader_uuid: "f358c50f28684a878bd82a66f0498a39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } health_report { overall_health: UNKNOWN } } }
I20260811 22:23:08.713763 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.710250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.29.3.66:39279)
0811 22:23:08.710419 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:08.710424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:08.710439 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:23:08.710524 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:08.710527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:08.710600 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:08.710709 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:23:08.710931 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.711465 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.712314 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:08.712530 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:08.712616 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:08.712917 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:23:08.713366 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:23:08.713477 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:08.713544 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 22:23:08.713609 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:23:08.720381 23032 raft_consensus.cc:1261] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Refusing update from remote peer f358c50f28684a878bd82a66f0498a39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:08.720382 23486 raft_consensus.cc:1261] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Refusing update from remote peer f358c50f28684a878bd82a66f0498a39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:08.721341 23608 consensus_queue.cc:1035] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:08.721571 23591 consensus_queue.cc:1035] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:08.739418 22794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:08.739645 22794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:08.742496 22794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Sending DeleteTablet for 3 replicas of tablet d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:08.743443 22995 tablet_service.cc:1514] Processing DeleteTablet for tablet d826229fbcc9485fa366539c1e474fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:08 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:32978
I20260811 22:23:08.743379 23226 tablet_service.cc:1514] Processing DeleteTablet for tablet d826229fbcc9485fa366539c1e474fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:08 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44412
I20260811 22:23:08.743382 23444 tablet_service.cc:1514] Processing DeleteTablet for tablet d826229fbcc9485fa366539c1e474fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:08 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:33736
I20260811 22:23:08.744014 23631 tablet_replica.cc:330] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: stopping tablet replica
I20260811 22:23:08.744088 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 22853
I20260811 22:23:08.754755 23631 raft_consensus.cc:2229] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:23:08.755492 23631 raft_consensus.cc:2258] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:08.757788 23631 ts_tablet_manager.cc:1905] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:08.760576 23631 ts_tablet_manager.cc:1918] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:08.760701 23631 log.cc:1196] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:08.760989 23631 ts_tablet_manager.cc:1939] T d826229fbcc9485fa366539c1e474fd9 P f358c50f28684a878bd82a66f0498a39: Deleting consensus metadata
I20260811 22:23:08.762326 22772 catalog_manager.cc:4841] TS f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): tablet d826229fbcc9485fa366539c1e474fd9 (table test-table [id=29092a6db86b40bf97a1d18ec6652174]) successfully deleted
I20260811 22:23:08.762567 23632 tablet_replica.cc:330] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: stopping tablet replica
I20260811 22:23:08.762914 23632 raft_consensus.cc:2229] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:08.763252 23632 raft_consensus.cc:2258] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:08.764699 23632 ts_tablet_manager.cc:1905] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:08.766937 23632 ts_tablet_manager.cc:1918] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:08.767055 23632 log.cc:1196] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:08.767369 23632 ts_tablet_manager.cc:1939] T d826229fbcc9485fa366539c1e474fd9 P 6e0326d3fa8a4381841179254b45cc6e: Deleting consensus metadata
I20260811 22:23:08.768616 22773 catalog_manager.cc:4841] TS 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): tablet d826229fbcc9485fa366539c1e474fd9 (table test-table [id=29092a6db86b40bf97a1d18ec6652174]) successfully deleted
W20260811 22:23:08.769096 22774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:44655 (error 108)
I20260811 22:23:08.769753 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 23111
I20260811 22:23:08.783416 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:08.769957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44655 (local address 127.0.0.1:32994)
0811 22:23:08.783165 (+ 13208us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:08.783296 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:44655: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13055}
W20260811 22:23:08.783705 22774 catalog_manager.cc:4584] TS 861058085af44412b85f3256353ea8b4 (127.29.3.65:44655): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d826229fbcc9485fa366539c1e474fd9: Network error: Client connection negotiation failed: client connection to 127.29.3.65:44655: connect: Connection refused (error 111)
I20260811 22:23:08.790060 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 23340
I20260811 22:23:08.809290 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 22742
I20260811 22:23:08.825277 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39005
--webserver_interface=127.29.3.126
--webserver_port=39965
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:39005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:08.943233 23634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:08.943521 23634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:08.943570 23634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:08.947722 23634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:08.947800 23634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:08.947820 23634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:08.947836 23634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:08.947850 23634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:08.953491 23634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:39005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:39005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=39965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:08.954710 23634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:08.955780 23634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:08.964455 23642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.964448 23641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:08.964448 23645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:08.964556 23634 server_base.cc:1034] running on GCE node
I20260811 22:23:08.965413 23634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:08.966681 23634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:08.967881 23634 hybrid_clock.cc:648] HybridClock initialized: now 1786486988967862 us; error 36 us; skew 500 ppm
Aug 11 22:23:08 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486988, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:08.971077 23634 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:08.972204 23634 webserver.cc:460] Webserver started at http://127.29.3.126:39965/ using document root <none> and password file <none>
I20260811 22:23:08.972816 23634 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:08.972891 23634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:08.977017 23634 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:23:08.979499 23657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:08.980885 23634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:08.981025 23634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "cce39c1cd1104ed3a181790c0eae3276"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "2b0a93c501096d2982e8a5c77b7da961"
server_key_iv: "97dedf1f312e9d8b959ebf24e58f2bb2"
server_key_version: "encryptionkey@0"
I20260811 22:23:08.981606 23634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:09.001147 23634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:09.004557 23634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:09.004765 23634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:09.012042 23634 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:39005
I20260811 22:23:09.012341 23745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:39005 every 8 connection(s)
I20260811 22:23:09.013250 23634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:09.017825 23747 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:23:09.019750 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap starting.
I20260811 22:23:09.023020 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 23634
I20260811 22:23:09.023526 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:44655
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:39005
--webserver_port=33841
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:09.043999 23747 log.cc:824] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:09.059283 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:23:09.059772 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276: Bootstrap complete.
I20260811 22:23:09.065442 23747 raft_consensus.cc:348] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:09.065986 23747 raft_consensus.cc:729] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cce39c1cd1104ed3a181790c0eae3276, State: Initialized, Role: FOLLOWER
I20260811 22:23:09.066610 23747 consensus_queue.cc:260] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:09.066766 23747 raft_consensus.cc:388] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:23:09.066838 23747 raft_consensus.cc:482] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:23:09.066937 23747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 2 FOLLOWER]: Advancing to term 3
I20260811 22:23:09.069689 23747 raft_consensus.cc:504] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:09.070093 23747 leader_election.cc:302] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cce39c1cd1104ed3a181790c0eae3276; no voters:
I20260811 22:23:09.070499 23747 leader_election.cc:288] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 22:23:09.070640 23751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 22:23:09.070923 23751 raft_consensus.cc:686] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [term 3 LEADER]: Becoming Leader. State: Replica: cce39c1cd1104ed3a181790c0eae3276, State: Running, Role: LEADER
I20260811 22:23:09.071290 23751 consensus_queue.cc:237] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } }
I20260811 22:23:09.071882 23747 sys_catalog.cc:564] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:09.072815 23756 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cce39c1cd1104ed3a181790c0eae3276. Latest consensus state: current_term: 3 leader_uuid: "cce39c1cd1104ed3a181790c0eae3276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } } }
I20260811 22:23:09.072966 23756 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:09.072837 23755 sys_catalog.cc:455] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cce39c1cd1104ed3a181790c0eae3276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cce39c1cd1104ed3a181790c0eae3276" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 39005 } } }
I20260811 22:23:09.073331 23770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:09.073350 23755 sys_catalog.cc:458] T 00000000000000000000000000000000 P cce39c1cd1104ed3a181790c0eae3276 [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:09.078817 23770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:09.079582 23770 catalog_manager.cc:1259] Loaded cluster ID: 6733f95e016d4ccc8ca7c87201751c46
I20260811 22:23:09.079654 23770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 22:23:09.080830 23770 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f525d01f9ad google::LogMessage::Fail() at ??:0
@ 0x7f525d02288a google::LogMessage::SendToLog() at ??:0
@ 0x7f525d01f4de google::LogMessage::Flush() at ??:0
@ 0x7f525d020a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f526795efdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f526795fd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f526795c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f52679a0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f525f2713d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f525f271c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f525f2733f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f525f262512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f52608a66db start_thread at ??:0
@ 0x7f525bb1d61f clone at ??:0
W20260811 22:23:09.170934 23749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:09.171231 23749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:09.171296 23749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:09.175444 23749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:09.175513 23749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:09.175598 23749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:23:09.181106 23749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:44655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=33841
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:09.182356 23749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:09.183470 23749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:09.193676 23811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:09.193848 23749 server_base.cc:1034] running on GCE node
W20260811 22:23:09.193673 23814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:09.193667 23816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:09.194578 23749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:09.195258 23749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:09.196435 23749 hybrid_clock.cc:648] HybridClock initialized: now 1786486989196404 us; error 45 us; skew 500 ppm
Aug 11 22:23:09 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486989, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:09.199342 23749 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:23:09.200289 23749 webserver.cc:460] Webserver started at http://127.29.3.65:33841/ using document root <none> and password file <none>
I20260811 22:23:09.200855 23749 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:09.200917 23749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:09.205214 23749 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:23:09.207670 23827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.209174 23749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:23:09.209303 23749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "861058085af44412b85f3256353ea8b4"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "f97c18368d8681e647f760a16f7a802d"
server_key_iv: "819d087d4dc4869d819c307693fa6906"
server_key_version: "encryptionkey@0"
I20260811 22:23:09.209896 23749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:09.220503 23749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:09.224023 23749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:09.224210 23749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:09.224869 23749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:09.226069 23840 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 22:23:09.231936 23749 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 22:23:09.232039 23749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260811 22:23:09.232115 23749 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 22:23:09.234421 23749 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 22:23:09.234491 23749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260811 22:23:09.235246 23840 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Bootstrap starting.
I20260811 22:23:09.245157 23749 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:44655
I20260811 22:23:09.245191 24012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:44655 every 8 connection(s)
I20260811 22:23:09.246209 23749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:23:09.253852 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 23749
I20260811 22:23:09.254359 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:39279
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:39005
--webserver_port=37117
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:09.259519 23840 log.cc:824] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:09.269930 23840 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:23:09.270479 23840 tablet_bootstrap.cc:492] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Bootstrap complete.
I20260811 22:23:09.272143 23840 ts_tablet_manager.cc:1397] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.004s
I20260811 22:23:09.275779 23840 raft_consensus.cc:348] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:09.276373 23840 raft_consensus.cc:729] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861058085af44412b85f3256353ea8b4, State: Initialized, Role: FOLLOWER
I20260811 22:23:09.276970 23840 consensus_queue.cc:260] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:09.278142 23840 ts_tablet_manager.cc:1428] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260811 22:23:09.323293 24018 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:23:09.248319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:46357)
0811 22:23:09.248843 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.248898 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:09.249743 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:09.322237 (+ 72494us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":140,"threads_started":1}
W20260811 22:23:09.323869 24014 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:39005 (0 consecutive failures): Network error: Failed to ping master at 127.29.3.126:39005: Client connection negotiation failed: client connection to 127.29.3.126:39005: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 22:23:09.324688 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.324374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:55377)
0811 22:23:09.324518 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.324612 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:23:09.325331 24018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.325107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:39925)
0811 22:23:09.325210 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.325274 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 22:23:09.325487 24014 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:23:09.375301 24019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:09.375582 24019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:09.375627 24019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:09.379681 24019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:09.379743 24019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:09.379801 24019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:23:09.385008 24019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:39279
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=37117
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:09.386057 24019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:09.387137 24019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:09.397338 24027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:09.397281 24029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:09.397441 24019 server_base.cc:1034] running on GCE node
W20260811 22:23:09.397281 24032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:09.397986 24019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:09.398685 24019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:09.399850 24019 hybrid_clock.cc:648] HybridClock initialized: now 1786486989399832 us; error 31 us; skew 500 ppm
Aug 11 22:23:09 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486989, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:09.402633 24019 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:23:09.403527 24019 webserver.cc:460] Webserver started at http://127.29.3.66:37117/ using document root <none> and password file <none>
I20260811 22:23:09.404119 24019 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:09.404187 24019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:09.408356 24019 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:09.410585 24045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.411974 24019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:23:09.412109 24019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "f358c50f28684a878bd82a66f0498a39"
format_stamp: "Formatted at 2026-08-11 22:23:06 on dist-test-slave-4tdj"
server_key: "dda47d301bad079ad70db636d6a0698d"
server_key_iv: "c18e8a0765cce04b8e8e2b60c9337201"
server_key_version: "encryptionkey@0"
I20260811 22:23:09.412686 24019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:09.428881 24019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:09.431902 24019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:09.432108 24019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:09.432749 24019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:09.433640 24019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:09.433691 24019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.433745 24019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:09.433772 24019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.442695 24019 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:39279
I20260811 22:23:09.442731 24245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:39279 every 8 connection(s)
I20260811 22:23:09.443737 24019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:23:09.447904 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.445960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:43371)
0811 22:23:09.446475 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.446603 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":119,"threads_started":1}
W20260811 22:23:09.448352 24246 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:39005 (0 consecutive failures): Network error: Failed to ping master at 127.29.3.126:39005: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
I20260811 22:23:09.448438 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 24019
I20260811 22:23:09.448930 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:46427
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:39005
--webserver_port=39121
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:09.448999 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.448784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:50577)
0811 22:23:09.448892 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.448941 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260811 22:23:09.449610 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.449421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:33353)
0811 22:23:09.449515 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.449562 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 22:23:09.449783 24246 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:23:09.566113 24255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:09.566402 24255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:09.566448 24255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:09.570505 24255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:09.570580 24255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:09.570643 24255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:23:09.576107 24255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:46427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=39121
--tserver_master_addrs=127.29.3.126:39005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:09.577250 24255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:09.578313 24255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:09.588320 24266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:09.588496 24255 server_base.cc:1034] running on GCE node
W20260811 22:23:09.588326 24270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:09.588313 24264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:09.589103 24255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:09.589773 24255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:09.590930 24255 hybrid_clock.cc:648] HybridClock initialized: now 1786486989590893 us; error 52 us; skew 500 ppm
Aug 11 22:23:09 dist-test-slave-4tdj krb5kdc[21907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486989, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:09.593623 24255 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:23:09.594544 24255 webserver.cc:460] Webserver started at http://127.29.3.67:39121/ using document root <none> and password file <none>
I20260811 22:23:09.595110 24255 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:09.595178 24255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:09.599157 24255 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:09.601326 24279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.602627 24255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:23:09.602768 24255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "6e0326d3fa8a4381841179254b45cc6e"
format_stamp: "Formatted at 2026-08-11 22:23:07 on dist-test-slave-4tdj"
server_key: "447b4355dd489f746bc53b71d156454e"
server_key_iv: "05af7cd61ba12dd794ea30b93e192659"
server_key_version: "encryptionkey@0"
I20260811 22:23:09.603336 24255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:09.633843 24255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:09.637260 24255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:09.637465 24255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:09.638128 24255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:09.639068 24255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:09.639122 24255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.639178 24255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:09.639209 24255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:09.648373 24255 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:46427
I20260811 22:23:09.648397 24492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:46427 every 8 connection(s)
I20260811 22:23:09.649427 24255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:23:09.653488 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.651688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:42605)
0811 22:23:09.652180 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.652304 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":129,"threads_started":1}
W20260811 22:23:09.654011 24496 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:39005 (0 consecutive failures): Network error: Failed to ping master at 127.29.3.126:39005: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
I20260811 22:23:09.654644 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.654431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:41367)
0811 22:23:09.654535 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.654586 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260811 22:23:09.655191 24499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:09.655021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:57731)
0811 22:23:09.655102 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:09.655141 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 22:23:09.655387 24496 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:23:09.656843 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 24255
I20260811 22:23:10.326669 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.326183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:38271)
0811 22:23:10.326474 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:10.326549 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":106,"threads_started":1}
I20260811 22:23:10.450117 24503 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:10.450377 24503 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:10.450932 24506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.450463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:54957)
0811 22:23:10.450761 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:10.450822 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":59,"threads_started":1}
I20260811 22:23:10.451530 24503 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
Aug 11 22:23:10 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486989, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.66@KRBTEST.COM
Aug 11 22:23:10 dist-test-slave-4tdj krb5kdc[21907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786486989, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.67@KRBTEST.COM
I20260811 22:23:10.470950 24510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.451800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39279 (local address 127.29.3.65:42363)
0811 22:23:10.452257 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:10.452269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:10.452358 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:10.453775 (+ 1417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:10.453782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:10.454476 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:10.455119 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:10.455138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.456429 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.456433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:10.457448 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:10.457459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.458215 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.458219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:10.458457 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:10.463756 (+ 5299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:10.463781 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:10.465739 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:10.468256 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:10.468263 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:10.468279 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:10.468652 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:10.469028 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:10.469031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:10.469033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:10.469186 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:10.469864 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:10.469871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:10.470133 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 22:23:10.470348 (+ 215us) client_negotiation.cc:240] Negotiation successful
0811 22:23:10.470762 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":316,"spinlock_wait_cycles":45184,"thread_start_us":75,"threads_started":1}
I20260811 22:23:10.470964 24509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.451918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:42363 (local address 127.29.3.66:39279)
0811 22:23:10.452300 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:10.452320 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:10.452359 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:23:10.453062 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:10.453067 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:10.453400 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:10.453704 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:10.455290 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.456271 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.457614 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.458084 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.458250 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:10.466122 (+ 7872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:10.466130 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:10.466147 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:10.466186 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:10.468097 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:10.468783 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:10.468789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:10.468794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:10.468926 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:10.469275 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:10.469279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:10.469280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:10.469742 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:10.469875 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:10.470474 (+ 599us) server_negotiation.cc:299] Negotiation successful
0811 22:23:10.470762 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":80,"threads_started":1}
I20260811 22:23:10.470963 24512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.451938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38981 (local address 127.29.3.67:46427)
0811 22:23:10.452317 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:10.452331 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:10.452372 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:23:10.453048 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:10.453054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:10.453400 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:10.453683 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:10.455293 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.456289 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.457575 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.458073 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.458224 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:10.466228 (+ 8004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:10.466234 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:10.466248 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:10.466282 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:10.468178 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:10.468788 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:10.468794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:10.468798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:10.468929 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:10.469270 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:10.469273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:10.469274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:10.469741 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:10.469875 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:10.470474 (+ 599us) server_negotiation.cc:299] Negotiation successful
0811 22:23:10.470763 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260811 22:23:10.470990 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.451789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:46427 (local address 127.29.3.65:38981)
0811 22:23:10.451934 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:10.451950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:10.452067 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:10.453775 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:10.453782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:10.454473 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:10.455120 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:10.455138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.456431 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.456433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:10.457446 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:10.457458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:10.458200 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:10.458204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:10.458458 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:10.463758 (+ 5300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:10.463782 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:10.465980 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:10.468324 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:10.468328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:10.468330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:10.468665 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:10.469024 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:10.469027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:10.469029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:10.469179 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:10.469864 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:10.469871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:10.470134 (+ 263us) client_negotiation.cc:769] Sending connection context
0811 22:23:10.470348 (+ 214us) client_negotiation.cc:240] Negotiation successful
0811 22:23:10.470846 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":378,"spinlock_wait_cycles":52992}
W20260811 22:23:10.474054 23829 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
W20260811 22:23:10.474272 23831 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:10.474412 23831 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 861058085af44412b85f3256353ea8b4; no voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39
I20260811 22:23:10.474699 24503 raft_consensus.cc:2726] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:23:10.656365 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:10.655954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:55191)
0811 22:23:10.656207 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:10.656265 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260811 22:23:11.328162 24515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:11.327613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:38743)
0811 22:23:11.327971 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:11.328052 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":86,"threads_started":1}
I20260811 22:23:11.452100 24516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:11.451676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:52537)
0811 22:23:11.451940 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:11.451997 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":87,"threads_started":1}
I20260811 22:23:11.657754 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:11.657227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:51833)
0811 22:23:11.657523 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:11.657626 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":87,"threads_started":1}
I20260811 22:23:12.329629 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:12.329102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:32775)
0811 22:23:12.329422 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:12.329490 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":96,"threads_started":1}
I20260811 22:23:12.453533 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:12.453026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:60059)
0811 22:23:12.453358 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:12.453414 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":122,"threads_started":1}
I20260811 22:23:12.504762 24523 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:12.504914 24523 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:12.505364 24523 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
W20260811 22:23:12.506254 23829 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
W20260811 22:23:12.506400 23831 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:12.506474 23831 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 861058085af44412b85f3256353ea8b4; no voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39
I20260811 22:23:12.506641 24523 raft_consensus.cc:2726] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:23:12.659066 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:12.658622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:37127)
0811 22:23:12.658904 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:12.658967 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260811 22:23:13.331032 24526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:13.330570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:37201)
0811 22:23:13.330857 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:13.330929 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
I20260811 22:23:13.454923 24527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:13.454456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:53391)
0811 22:23:13.454750 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:13.454820 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260811 22:23:13.660321 24528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:13.659860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:35139)
0811 22:23:13.660141 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:13.660206 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260811 22:23:14.332484 24529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:14.331995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:60067)
0811 22:23:14.332314 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:14.332380 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
I20260811 22:23:14.456284 24531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:14.455786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:58139)
0811 22:23:14.456099 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:14.456168 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260811 22:23:14.661561 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:14.661102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:40757)
0811 22:23:14.661402 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:14.661461 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":101,"threads_started":1}
I20260811 22:23:15.127297 24533 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:15.127439 24533 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:15.127900 24533 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
W20260811 22:23:15.128855 23831 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
W20260811 22:23:15.128978 23829 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:15.129043 23829 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 861058085af44412b85f3256353ea8b4; no voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39
I20260811 22:23:15.129185 24533 raft_consensus.cc:2726] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:23:15.333696 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:15.333267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:41747)
0811 22:23:15.333534 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:15.333592 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":77,"threads_started":1}
I20260811 22:23:15.457566 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:15.457100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:45493)
0811 22:23:15.457399 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:15.457465 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260811 22:23:15.662968 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:15.662466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:38859)
0811 22:23:15.662800 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:15.662867 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260811 22:23:16.335124 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:16.334615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:52505)
0811 22:23:16.334947 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:16.335016 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260811 22:23:16.458967 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:16.458462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:52459)
0811 22:23:16.458805 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:16.458866 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":76,"threads_started":1}
I20260811 22:23:16.664433 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:16.663894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:48163)
0811 22:23:16.664264 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:16.664330 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":84,"threads_started":1}
I20260811 22:23:17.336447 24545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:17.335977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:36963)
0811 22:23:17.336265 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:17.336334 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260811 22:23:17.460337 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:17.459845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:39655)
0811 22:23:17.460167 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:17.460234 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260811 22:23:17.665670 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:17.665200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:49975)
0811 22:23:17.665506 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:17.665568 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260811 22:23:18.337828 24551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:18.337292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:50577)
0811 22:23:18.337598 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:18.337716 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
I20260811 22:23:18.461742 24552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:18.461242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:35841)
0811 22:23:18.461535 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:18.461612 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":80,"threads_started":1}
I20260811 22:23:18.666887 24553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:18.666477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:38841)
0811 22:23:18.666730 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:18.666786 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":79,"threads_started":1}
I20260811 22:23:19.226267 24554 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:19.226455 24554 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:19.226914 24554 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
W20260811 22:23:19.227742 23829 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
W20260811 22:23:19.227886 23831 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:19.227978 23831 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 861058085af44412b85f3256353ea8b4; no voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39
I20260811 22:23:19.228117 24554 raft_consensus.cc:2726] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:23:19.339326 24556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:19.338858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:39087)
0811 22:23:19.339133 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:19.339213 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260811 22:23:19.463112 24557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:19.462604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:42597)
0811 22:23:19.462934 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:19.463010 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":108,"threads_started":1}
I20260811 22:23:19.668388 24558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:19.667874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:59747)
0811 22:23:19.668189 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:19.668265 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20260811 22:23:20.340883 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:20.340311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:39181)
0811 22:23:20.340688 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:20.340763 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":87,"threads_started":1}
I20260811 22:23:20.464489 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:20.463989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:54953)
0811 22:23:20.464310 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:20.464382 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":87,"threads_started":1}
I20260811 22:23:20.669731 24561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:20.669257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:49101)
0811 22:23:20.669561 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:20.669628 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260811 22:23:21.342233 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:21.341811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:52545)
0811 22:23:21.342073 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:21.342130 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260811 22:23:21.465818 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:21.465357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:44155)
0811 22:23:21.465630 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:21.465711 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260811 22:23:21.671069 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:21.670604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:43483)
0811 22:23:21.670894 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:21.670959 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
I20260811 22:23:22.343616 24574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:22.343127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:54917)
0811 22:23:22.343434 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:22.343511 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260811 22:23:22.467177 24576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:22.466684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:35119)
0811 22:23:22.466994 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:22.467073 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260811 22:23:22.672367 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:22.671921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:58907)
0811 22:23:22.672200 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:22.672265 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260811 22:23:22.856864 24580 raft_consensus.cc:482] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:22.857005 24580 raft_consensus.cc:504] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e0326d3fa8a4381841179254b45cc6e" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 46427 } } peers { permanent_uuid: "f358c50f28684a878bd82a66f0498a39" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39279 } } peers { permanent_uuid: "861058085af44412b85f3256353ea8b4" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44655 } }
I20260811 22:23:22.857465 24580 leader_election.cc:288] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427), f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279)
W20260811 22:23:22.858283 23831 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6e0326d3fa8a4381841179254b45cc6e (127.29.3.67:46427): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
W20260811 22:23:22.858438 23829 leader_election.cc:341] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f358c50f28684a878bd82a66f0498a39 (127.29.3.66:39279): Not found: Tablet not found: d826229fbcc9485fa366539c1e474fd9
I20260811 22:23:22.858531 23829 leader_election.cc:302] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 861058085af44412b85f3256353ea8b4; no voters: 6e0326d3fa8a4381841179254b45cc6e, f358c50f28684a878bd82a66f0498a39
I20260811 22:23:22.858698 24580 raft_consensus.cc:2726] T d826229fbcc9485fa366539c1e474fd9 P 861058085af44412b85f3256353ea8b4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:23:23.345021 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:23.344511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:47377)
0811 22:23:23.344848 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:23.344915 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260811 22:23:23.468621 24583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:23.468095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:34579)
0811 22:23:23.468414 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:23.468509 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":90,"threads_started":1}
I20260811 22:23:23.673714 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:23.673226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:47775)
0811 22:23:23.673536 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:23.673608 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
I20260811 22:23:24.346629 24585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:24.346078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.65:37219)
0811 22:23:24.346428 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:24.346495 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260811 22:23:24.470175 24588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:24.469637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.66:54743)
0811 22:23:24.469998 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:24.470071 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260811 22:23:24.660353 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 23749
I20260811 22:23:24.700379 24589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:24.699902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:39005 (local address 127.29.3.67:42609)
0811 22:23:24.700220 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:24.700278 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:39005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":72,"threads_started":1}
I20260811 22:23:24.705446 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 24019
I20260811 22:23:24.724710 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 24255
2026-08-11T22:23:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19827 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:23:24 dist-test-slave-4tdj krb5kdc[24594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:23:24 dist-test-slave-4tdj krb5kdc[24594](info): set up 2 sockets
Aug 11 22:23:24 dist-test-slave-4tdj krb5kdc[24594](info): commencing operation
krb5kdc: starting...
W20260811 22:23:25.929076 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:23:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:23:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:23:26.067404 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38759
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:26.184000 24616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:26.184304 24616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:26.184348 24616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:26.188529 24616 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 22:23:26.188606 24616 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 22:23:26.188621 24616 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 22:23:26.188635 24616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:26.188652 24616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:26.188669 24616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:26.188683 24616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:26.188697 24616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:26.194437 24616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:26.195598 24616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:26.196611 24616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:26.206043 24626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:26.206161 24616 server_base.cc:1034] running on GCE node
W20260811 22:23:26.206102 24624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:26.206089 24630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:26.206951 24616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:26.208185 24616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:26.209359 24616 hybrid_clock.cc:648] HybridClock initialized: now 1786487006209345 us; error 40 us; skew 500 ppm
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:26.212363 24616 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:26.213382 24616 webserver.cc:460] Webserver started at http://127.29.3.126:44143/ using document root <none> and password file <none>
I20260811 22:23:26.213972 24616 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:26.214043 24616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:26.214318 24616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:26.216512 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "b62badc91ad247c1bf5d18c8fe4563ae"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "8a269d9d89fe21ea1b49f8b2f58b4fb6"
server_key_iv: "9ee137dda81e2c8d478bd5507aa56424"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.217089 24616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "b62badc91ad247c1bf5d18c8fe4563ae"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "8a269d9d89fe21ea1b49f8b2f58b4fb6"
server_key_iv: "9ee137dda81e2c8d478bd5507aa56424"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.221057 24616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:23:26.223727 24641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.225173 24616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:23:26.225319 24616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "b62badc91ad247c1bf5d18c8fe4563ae"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "8a269d9d89fe21ea1b49f8b2f58b4fb6"
server_key_iv: "9ee137dda81e2c8d478bd5507aa56424"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.225430 24616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:26.247351 24616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:26.250420 24616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:26.250617 24616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:26.258201 24616 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:38759
I20260811 22:23:26.258502 24743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:38759 every 8 connection(s)
I20260811 22:23:26.259342 24616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:26.262428 24745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:26.265343 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 24616
I20260811 22:23:26.265448 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:23:26.265733 29709 external_mini_cluster.cc:1428] Setting key a00cb7b7a3d40bc03163d298dfa1659c
I20260811 22:23:26.268805 24745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap starting.
I20260811 22:23:26.271198 24745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:26.271986 24745 log.cc:824] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:26.274449 24745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: No bootstrap required, opened a new log
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:26.279246 24745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:26.279605 24745 raft_consensus.cc:374] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:26.279677 24745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62badc91ad247c1bf5d18c8fe4563ae, State: Initialized, Role: FOLLOWER
I20260811 22:23:26.280220 24745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:26.280385 24745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:23:26.280450 24745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:23:26.280546 24745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:26.280841 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:26.267224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52334 (local address 127.29.3.126:38759)
0811 22:23:26.267728 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:26.267744 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:26.267781 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:23:26.268448 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:26.268467 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:26.268803 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:26.269143 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:26.269928 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.270996 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.271908 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.272458 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.272621 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:26.275363 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:26.275370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:26.275389 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:26.275424 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:26.277351 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:26.277903 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:26.277910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:26.277917 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:26.278076 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:26.278374 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:26.278377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:26.278379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:26.278956 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:26.279182 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:26.279435 (+ 253us) server_negotiation.cc:299] Negotiation successful
0811 22:23:26.279777 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":162,"threads_started":1}
I20260811 22:23:26.281646 24745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:26.282022 24745 leader_election.cc:302] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b62badc91ad247c1bf5d18c8fe4563ae; no voters:
I20260811 22:23:26.282439 24745 leader_election.cc:288] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:23:26.282567 24753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:26.282799 24753 raft_consensus.cc:686] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 LEADER]: Becoming Leader. State: Replica: b62badc91ad247c1bf5d18c8fe4563ae, State: Running, Role: LEADER
I20260811 22:23:26.283192 24753 consensus_queue.cc:237] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:26.283727 24745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:26.284864 24754 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } } }
I20260811 22:23:26.285001 24754 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:26.285302 24756 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b62badc91ad247c1bf5d18c8fe4563ae. Latest consensus state: current_term: 1 leader_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } } }
I20260811 22:23:26.285399 24756 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:26.285410 24769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:26.288622 24769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:26.295060 24769 catalog_manager.cc:1347] Generated new cluster ID: a4bac1c3a8564a43bfe2fce718af424e
I20260811 22:23:26.295153 24769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:23:26.309849 24769 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:23:26.311352 24769 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:23:26.317744 24769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Generated new TSK 0
I20260811 22:23:26.318718 24769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:23:26.351763 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38759
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:26.471086 24790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:26.471386 24790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:26.471450 24790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:26.475636 24790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:26.475711 24790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:26.475795 24790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:23:26.481359 24790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:26.482549 24790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:26.483608 24790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:26.493918 24798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:26.493939 24797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:26.494051 24790 server_base.cc:1034] running on GCE node
W20260811 22:23:26.493929 24801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:26.494598 24790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:26.495278 24790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:26.496441 24790 hybrid_clock.cc:648] HybridClock initialized: now 1786487006496421 us; error 30 us; skew 500 ppm
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:26.499208 24790 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:23:26.500111 24790 webserver.cc:460] Webserver started at http://127.29.3.65:40049/ using document root <none> and password file <none>
I20260811 22:23:26.500644 24790 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:26.500712 24790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:26.500886 24790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:26.503204 24790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "8f3055bb4c254788a73c91ddc8ee5e92"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "03d59ff313ea61c5f811cd9941be2ee8"
server_key_iv: "d9421c3fec00d70c75b26ee603f3c5a1"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.503741 24790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "8f3055bb4c254788a73c91ddc8ee5e92"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "03d59ff313ea61c5f811cd9941be2ee8"
server_key_iv: "d9421c3fec00d70c75b26ee603f3c5a1"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.507493 24790 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260811 22:23:26.510023 24811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.511298 24790 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:23:26.511409 24790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "8f3055bb4c254788a73c91ddc8ee5e92"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "03d59ff313ea61c5f811cd9941be2ee8"
server_key_iv: "d9421c3fec00d70c75b26ee603f3c5a1"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.511525 24790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:26.523391 24790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:26.526726 24790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:26.526907 24790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:26.527530 24790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:26.528414 24790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:26.528465 24790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.528519 24790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:26.528544 24790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.538048 24790 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:35483
I20260811 22:23:26.538141 25000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:35483 every 8 connection(s)
I20260811 22:23:26.539050 24790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:23:26.539907 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 24790
I20260811 22:23:26.540022 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:23:26.540329 29709 external_mini_cluster.cc:1428] Setting key 29ffb5d939c04befd23be7b36b9404c2
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:23:26.554207 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:26.541107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:43149 (local address 127.29.3.126:38759)
0811 22:23:26.541244 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:26.541248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:26.542328 (+ 1080us) server_negotiation.cc:407] Connection header received
0811 22:23:26.543724 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:26.543728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:26.543789 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:26.543866 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:26.545819 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.546435 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.547644 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.547886 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.547976 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:26.550772 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:26.550776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:26.550778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:26.550805 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:26.552167 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:26.552671 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:26.552674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:26.552675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:26.552755 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:26.553077 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:26.553080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:26.553081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:26.553277 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:26.553365 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:26.553899 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 22:23:26.554034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:23:26.555183 25005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:26.541611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:43149)
0811 22:23:26.542125 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:26.542181 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:26.543496 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:26.544473 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:26.544495 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:26.545008 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:26.545654 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:26.545671 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.546583 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.546586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:26.547506 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:26.547518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.548041 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.548046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:26.548237 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:26.548873 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:26.548891 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:26.550581 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:26.552287 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:26.552294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:26.552312 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:26.552569 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:26.552840 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:26.552844 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:26.552846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:26.552988 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:26.553380 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:26.553388 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:26.553658 (+ 270us) client_negotiation.cc:769] Sending connection context
0811 22:23:26.553838 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 22:23:26.554201 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":119,"threads_started":1}
I20260811 22:23:26.556532 25001 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38759
I20260811 22:23:26.556774 25001 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:26.557314 25001 heartbeater.cc:511] Master 127.29.3.126:38759 requested a full tablet report, sending...
I20260811 22:23:26.559010 24684 ts_manager.cc:194] Registered new tserver with Master: 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483)
I20260811 22:23:26.560671 24684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:43149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:23:26.571499 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38759
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:26.690050 25009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:26.690407 25009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:26.690488 25009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:26.694698 25009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:26.694777 25009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:26.694862 25009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:23:26.700469 25009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:26.701627 25009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:26.702777 25009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:26.713078 25020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:26.713078 25018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:26.713121 25022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:26.713290 25009 server_base.cc:1034] running on GCE node
I20260811 22:23:26.713876 25009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:26.714663 25009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:26.715849 25009 hybrid_clock.cc:648] HybridClock initialized: now 1786487006715831 us; error 29 us; skew 500 ppm
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:26.719110 25009 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:23:26.720101 25009 webserver.cc:460] Webserver started at http://127.29.3.66:39893/ using document root <none> and password file <none>
I20260811 22:23:26.720682 25009 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:26.720754 25009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:26.720986 25009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:26.723428 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "532715aa46b244a2a96f3cc396e7eef3"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "4e007db31875c9d8f94556ee8d65ada9"
server_key_iv: "70a275fe8432f4dc9cd5fe918e4b4e4f"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.724030 25009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "532715aa46b244a2a96f3cc396e7eef3"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "4e007db31875c9d8f94556ee8d65ada9"
server_key_iv: "70a275fe8432f4dc9cd5fe918e4b4e4f"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.728248 25009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:23:26.731189 25033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.732623 25009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:23:26.732774 25009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "532715aa46b244a2a96f3cc396e7eef3"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "4e007db31875c9d8f94556ee8d65ada9"
server_key_iv: "70a275fe8432f4dc9cd5fe918e4b4e4f"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.732897 25009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:26.745322 25009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:26.748471 25009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:26.748683 25009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:26.749368 25009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:26.750319 25009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:26.750379 25009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.750437 25009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:26.750461 25009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.759289 25009 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:45681
I20260811 22:23:26.759335 25217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:45681 every 8 connection(s)
I20260811 22:23:26.760236 25009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:23:26.769435 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 25009
I20260811 22:23:26.769574 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:23:26.769857 29709 external_mini_cluster.cc:1428] Setting key 642a5799325fe3f2d36f7cc4a74f8783
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:26.774679 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:26.762101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:43859 (local address 127.29.3.126:38759)
0811 22:23:26.762253 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:26.762258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:26.763258 (+ 1000us) server_negotiation.cc:407] Connection header received
0811 22:23:26.764528 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:26.764531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:26.764589 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:26.764666 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:26.765929 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.766546 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.767694 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.767913 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.767984 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:26.770817 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:26.770820 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:26.770822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:26.770850 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:26.772410 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:26.772998 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:26.773001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:26.773003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:26.773093 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:26.773464 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:26.773467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:26.773468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:26.773678 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:26.773768 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:26.774366 (+ 598us) server_negotiation.cc:299] Negotiation successful
0811 22:23:26.774506 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:23:26.775590 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:26.762606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:43859)
0811 22:23:26.763082 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:26.763128 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:26.764305 (+ 1177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:26.764833 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:26.764839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:26.765263 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:26.765773 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:26.765788 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.766669 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.766671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:26.767586 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:26.767596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:26.768034 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:26.768036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:26.768200 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:26.768850 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:26.768868 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:26.770638 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:26.772549 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:26.772556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:26.772574 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:26.772874 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:26.773198 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:26.773200 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:26.773202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:26.773358 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:26.773785 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:26.773793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:26.774088 (+ 295us) client_negotiation.cc:769] Sending connection context
0811 22:23:26.774275 (+ 187us) client_negotiation.cc:240] Negotiation successful
0811 22:23:26.774698 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":121,"threads_started":1}
I20260811 22:23:26.776851 25218 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38759
I20260811 22:23:26.777158 25218 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:26.777829 25218 heartbeater.cc:511] Master 127.29.3.126:38759 requested a full tablet report, sending...
I20260811 22:23:26.779170 24684 ts_manager.cc:194] Registered new tserver with Master: 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681)
I20260811 22:23:26.779958 24684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:43859
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:23:26.802802 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38759
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:26.920001 25226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:26.920306 25226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:26.920372 25226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:26.924542 25226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:26.924620 25226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:26.924681 25226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:23:26.930378 25226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:26.931548 25226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:26.932578 25226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:26.942728 25239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:26.942728 25237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:26.942795 25236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:26.943558 25226 server_base.cc:1034] running on GCE node
I20260811 22:23:26.943979 25226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:26.944686 25226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:26.945868 25226 hybrid_clock.cc:648] HybridClock initialized: now 1786487006945844 us; error 37 us; skew 500 ppm
Aug 11 22:23:26 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:26.955844 25226 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:23:26.957162 25226 webserver.cc:460] Webserver started at http://127.29.3.67:37463/ using document root <none> and password file <none>
I20260811 22:23:26.957943 25226 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:26.958034 25226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:26.958320 25226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:23:26.960708 25226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "a5d58361fb004a7cb54d23491d52c821"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "09a350fa4725f962d72608ba2ba93962"
server_key_iv: "5eb9f2e38a5cc53e85200356efdd260e"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.961659 25226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "a5d58361fb004a7cb54d23491d52c821"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "09a350fa4725f962d72608ba2ba93962"
server_key_iv: "5eb9f2e38a5cc53e85200356efdd260e"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.967264 25226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260811 22:23:26.970868 25248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.972790 25226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:26.973003 25226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "a5d58361fb004a7cb54d23491d52c821"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "09a350fa4725f962d72608ba2ba93962"
server_key_iv: "5eb9f2e38a5cc53e85200356efdd260e"
server_key_version: "encryptionkey@0"
I20260811 22:23:26.973219 25226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:26.991488 25226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:26.995437 25226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:26.995658 25226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:26.996330 25226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:26.997303 25226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:26.997364 25226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:26.997419 25226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:26.997447 25226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:27.007491 25226 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:35367
I20260811 22:23:27.007542 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:35367 every 8 connection(s)
I20260811 22:23:27.008455 25226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:23:27.011442 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 25226
I20260811 22:23:27.011580 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:23:27.011885 29709 external_mini_cluster.cc:1428] Setting key 23897ad06d0fd348fd0c229001831348
Aug 11 22:23:27 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487006, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:27.023692 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.010683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:57235 (local address 127.29.3.126:38759)
0811 22:23:27.010850 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.010854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.011877 (+ 1023us) server_negotiation.cc:407] Connection header received
0811 22:23:27.013328 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.013331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.013390 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.013463 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:27.014826 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.015365 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.016510 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.016739 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.016827 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.019402 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:27.019405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:27.019408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:27.019439 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:27.021198 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:27.021855 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:27.021859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:27.021861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:27.021960 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:27.022376 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:27.022380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:27.022382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:27.022636 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:27.022746 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.023427 (+ 681us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.023561 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:23:27.024886 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.011123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:57235)
0811 22:23:27.011701 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.011760 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:27.013105 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:27.013645 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:27.013652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:27.014104 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:27.014683 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.014700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.015487 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.015490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.016390 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.016400 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.016880 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.016884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.017055 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.017584 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:27.017600 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:27.019196 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:27.021352 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:27.021361 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:27.021382 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:27.021742 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:27.022047 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:27.022050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:27.022051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:27.022257 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:27.022739 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:27.022746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:27.023098 (+ 352us) client_negotiation.cc:769] Sending connection context
0811 22:23:27.023334 (+ 236us) client_negotiation.cc:240] Negotiation successful
0811 22:23:27.023730 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":154,"threads_started":1}
I20260811 22:23:27.026407 25439 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38759
I20260811 22:23:27.026675 25439 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:27.027240 25439 heartbeater.cc:511] Master 127.29.3.126:38759 requested a full tablet report, sending...
I20260811 22:23:27.028309 24684 ts_manager.cc:194] Registered new tserver with Master: a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
I20260811 22:23:27.028898 24684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:57235
I20260811 22:23:27.038010 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:23:27.047646 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.040264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52336 (local address 127.29.3.126:38759)
0811 22:23:27.040458 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.040462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.040608 (+ 146us) server_negotiation.cc:407] Connection header received
0811 22:23:27.040765 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.040768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.040814 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.040882 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:27.041631 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.042167 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.043045 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.043273 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.043362 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.044718 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:27.044723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:27.044726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:27.044754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:27.045949 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:27.046391 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:27.046395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:27.046397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:27.046482 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:27.046736 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:27.046739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:27.046741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:27.046982 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:27.047107 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.047348 (+ 241us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.047471 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260811 22:23:27.050573 24684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:23:27.053732 24684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:23:27.069581 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.063411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35483 (local address 127.0.0.1:47786)
0811 22:23:27.064323 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.064381 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:27.064532 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:27.065346 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:27.065353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:27.065384 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:27.065781 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.065787 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.067155 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.067157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.068147 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.068157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.068989 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.068992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.069108 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.069149 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:23:27.069263 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 22:23:27.069398 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":278,"spinlock_wait_cycles":5120,"thread_start_us":73,"threads_started":1}
I20260811 22:23:27.069725 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.064096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.29.3.66:45681)
0811 22:23:27.065059 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.065072 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.065094 (+ 22us) server_negotiation.cc:407] Connection header received
0811 22:23:27.065166 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.065170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.065350 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.065523 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:27.065747 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.066958 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.068309 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.069088 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.069246 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.069371 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.069467 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.069570 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":855,"thread_start_us":100,"threads_started":1}
I20260811 22:23:27.069810 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.063776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:35367 (local address 127.0.0.1:33476)
0811 22:23:27.064718 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.064731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:27.064803 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:27.065347 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:27.065353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:27.065391 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:27.065634 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.065643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.066728 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.066731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.068101 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.068112 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.069413 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.069418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.069518 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.069542 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:23:27.069606 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:23:27.069686 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":51,"spinlock_wait_cycles":12416,"thread_start_us":117,"threads_started":1}
I20260811 22:23:27.069591 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.063559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45681 (local address 127.0.0.1:54886)
0811 22:23:27.064327 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.064383 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:27.064532 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:27.065514 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:27.065517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:27.065529 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:27.065631 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.065638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.067101 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.067104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.068210 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.068219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.069263 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.069267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.069349 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.069369 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 22:23:27.069434 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 22:23:27.069492 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":3,"spinlock_wait_cycles":6784,"thread_start_us":146,"threads_started":1}
I20260811 22:23:27.070230 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.064253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.29.3.67:35367)
0811 22:23:27.064823 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.064835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.064858 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:23:27.064932 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.064937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.065149 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.065457 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:27.065754 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.066608 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.068231 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.069088 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.069199 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.069347 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.070015 (+ 668us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.070111 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":145,"threads_started":1}
I20260811 22:23:27.070514 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.063574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47786 (local address 127.29.3.65:35483)
0811 22:23:27.064538 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.064546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.064571 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:23:27.064750 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.064757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.064986 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.065146 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:27.065900 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.067030 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.068273 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.068864 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.068968 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.069148 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.069674 (+ 526us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.070378 (+ 704us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":143,"threads_started":1}
I20260811 22:23:27.073179 25322 tablet_service.cc:1467] Processing CreateTablet for tablet 525e25e504374696bc860bb69eaf2380 (DEFAULT_TABLE table=test-table [id=f3034b52c32c4887a0823d6e60e0f558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:27.073364 25113 tablet_service.cc:1467] Processing CreateTablet for tablet 525e25e504374696bc860bb69eaf2380 (DEFAULT_TABLE table=test-table [id=f3034b52c32c4887a0823d6e60e0f558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:27.073540 24893 tablet_service.cc:1467] Processing CreateTablet for tablet 525e25e504374696bc860bb69eaf2380 (DEFAULT_TABLE table=test-table [id=f3034b52c32c4887a0823d6e60e0f558]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:27.074250 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525e25e504374696bc860bb69eaf2380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:27.074501 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525e25e504374696bc860bb69eaf2380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:27.074501 25113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525e25e504374696bc860bb69eaf2380. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:27.081231 25474 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Bootstrap starting.
I20260811 22:23:27.082617 25475 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap starting.
I20260811 22:23:27.082818 25476 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Bootstrap starting.
I20260811 22:23:27.084604 25475 tablet_bootstrap.cc:654] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:27.084636 25474 tablet_bootstrap.cc:654] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:27.085518 25476 tablet_bootstrap.cc:654] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:27.085490 25475 log.cc:824] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:27.085582 25474 log.cc:824] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:27.086427 25476 log.cc:824] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:27.088085 25476 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: No bootstrap required, opened a new log
I20260811 22:23:27.088129 25475 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: No bootstrap required, opened a new log
I20260811 22:23:27.088151 25474 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: No bootstrap required, opened a new log
I20260811 22:23:27.088295 25476 ts_tablet_manager.cc:1397] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:23:27.088315 25475 ts_tablet_manager.cc:1397] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:23:27.088315 25474 ts_tablet_manager.cc:1397] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:23:27.091529 25476 raft_consensus.cc:348] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.091807 25476 raft_consensus.cc:374] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:27.091864 25476 raft_consensus.cc:729] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532715aa46b244a2a96f3cc396e7eef3, State: Initialized, Role: FOLLOWER
I20260811 22:23:27.092324 25476 consensus_queue.cc:260] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.092209 25474 raft_consensus.cc:348] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.092479 25474 raft_consensus.cc:374] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:27.092546 25474 raft_consensus.cc:729] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d58361fb004a7cb54d23491d52c821, State: Initialized, Role: FOLLOWER
I20260811 22:23:27.092715 25475 raft_consensus.cc:348] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.092957 25475 raft_consensus.cc:374] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:27.093003 25475 raft_consensus.cc:729] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3055bb4c254788a73c91ddc8ee5e92, State: Initialized, Role: FOLLOWER
I20260811 22:23:27.092978 25474 consensus_queue.cc:260] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.093415 25475 consensus_queue.cc:260] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.093596 25476 ts_tablet_manager.cc:1428] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 22:23:27.093963 25474 ts_tablet_manager.cc:1428] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:23:27.093979 25218 heartbeater.cc:503] Master 127.29.3.126:38759 was elected leader, sending a full tablet report...
I20260811 22:23:27.094276 25475 ts_tablet_manager.cc:1428] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:23:27.094385 25439 heartbeater.cc:503] Master 127.29.3.126:38759 was elected leader, sending a full tablet report...
I20260811 22:23:27.094620 25001 heartbeater.cc:503] Master 127.29.3.126:38759 was elected leader, sending a full tablet report...
I20260811 22:23:27.176481 25483 raft_consensus.cc:482] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:27.176824 25483 raft_consensus.cc:504] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.178138 25483 leader_election.cc:288] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483), 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681)
I20260811 22:23:27.181886 25443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.178438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45681 (local address 127.29.3.67:42113)
0811 22:23:27.178551 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.178566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:27.178673 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:27.178987 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:27.178990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:27.179003 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:27.179123 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.179129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.179919 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.179922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.180770 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.180780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.181442 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.181446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.181558 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.181578 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 22:23:27.181646 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 22:23:27.181735 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":24}
I20260811 22:23:27.182026 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.178654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:60269 (local address 127.29.3.65:35483)
0811 22:23:27.178768 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.178773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.178976 (+ 203us) server_negotiation.cc:407] Connection header received
0811 22:23:27.179071 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.179074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.179132 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.179211 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:27.179504 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.179990 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.181033 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.181463 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.181557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.181619 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.181822 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.181899 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:23:27.182049 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.178554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:42113 (local address 127.29.3.66:45681)
0811 22:23:27.178745 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.178749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.178767 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:23:27.178829 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.178833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.178878 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.178964 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:27.179253 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.179800 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.180902 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.181317 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.181415 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.181484 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.181680 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.181759 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260811 22:23:27.182431 25489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.178437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35483 (local address 127.29.3.67:60269)
0811 22:23:27.178882 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.178893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:27.178985 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:27.179255 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:27.179259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:27.179278 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:27.179390 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.179396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.180100 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.180104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.180909 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:27.180919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.181588 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.181592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:27.181702 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.181721 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:23:27.181793 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 22:23:27.181866 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":154,"thread_start_us":152,"threads_started":1}
I20260811 22:23:27.182909 25140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525e25e504374696bc860bb69eaf2380" candidate_uuid: "a5d58361fb004a7cb54d23491d52c821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532715aa46b244a2a96f3cc396e7eef3" is_pre_election: true
I20260811 22:23:27.183044 24923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525e25e504374696bc860bb69eaf2380" candidate_uuid: "a5d58361fb004a7cb54d23491d52c821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" is_pre_election: true
I20260811 22:23:27.183302 24923 raft_consensus.cc:2454] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d58361fb004a7cb54d23491d52c821 in term 0.
I20260811 22:23:27.183303 25140 raft_consensus.cc:2454] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d58361fb004a7cb54d23491d52c821 in term 0.
I20260811 22:23:27.183862 25252 leader_election.cc:302] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92, a5d58361fb004a7cb54d23491d52c821; no voters:
I20260811 22:23:27.184201 25483 raft_consensus.cc:2781] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:23:27.184293 25483 raft_consensus.cc:482] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:23:27.184340 25483 raft_consensus.cc:3037] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:27.185563 25483 raft_consensus.cc:504] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.185979 25483 leader_election.cc:288] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [CANDIDATE]: Term 1 election: Requested vote from peers 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483), 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681)
I20260811 22:23:27.186386 24923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525e25e504374696bc860bb69eaf2380" candidate_uuid: "a5d58361fb004a7cb54d23491d52c821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3055bb4c254788a73c91ddc8ee5e92"
I20260811 22:23:27.186542 24923 raft_consensus.cc:3037] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:27.186635 25140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525e25e504374696bc860bb69eaf2380" candidate_uuid: "a5d58361fb004a7cb54d23491d52c821" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532715aa46b244a2a96f3cc396e7eef3"
I20260811 22:23:27.186779 25140 raft_consensus.cc:3037] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:27.187712 24923 raft_consensus.cc:2454] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d58361fb004a7cb54d23491d52c821 in term 1.
I20260811 22:23:27.187839 25140 raft_consensus.cc:2454] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d58361fb004a7cb54d23491d52c821 in term 1.
I20260811 22:23:27.188081 25252 leader_election.cc:302] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92, a5d58361fb004a7cb54d23491d52c821; no voters:
I20260811 22:23:27.188252 25483 raft_consensus.cc:2781] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:27.188485 25483 raft_consensus.cc:686] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 1 LEADER]: Becoming Leader. State: Replica: a5d58361fb004a7cb54d23491d52c821, State: Running, Role: LEADER
I20260811 22:23:27.188864 25483 consensus_queue.cc:237] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:27.192329 24683 catalog_manager.cc:5495] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d58361fb004a7cb54d23491d52c821 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "a5d58361fb004a7cb54d23491d52c821" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } health_report { overall_health: HEALTHY } } }
I20260811 22:23:27.224732 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.221422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.29.3.67:35367)
0811 22:23:27.221537 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:27.221540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:27.221554 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:23:27.221671 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:27.221673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:27.221734 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:27.221829 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:23:27.222035 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.222565 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.223374 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:27.223580 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:27.223663 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:27.223897 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:23:27.224340 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:23:27.224431 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:27.224510 (+ 79us) server_negotiation.cc:299] Negotiation successful
0811 22:23:27.224567 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:23:27.231022 25140 raft_consensus.cc:1261] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Refusing update from remote peer a5d58361fb004a7cb54d23491d52c821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:27.231022 24923 raft_consensus.cc:1261] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Refusing update from remote peer a5d58361fb004a7cb54d23491d52c821: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:27.231736 25490 consensus_queue.cc:1035] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:27.231932 25483 consensus_queue.cc:1035] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [LEADER]: Connected to new peer: Peer: permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:27.241376 25499 mvcc.cc:205] Tried to move back new op lower bound from 7317450781610291200 to 7317450781448445952. Current Snapshot: MvccSnapshot[applied={T|T < 7317450781610291200}]
I20260811 22:23:27.244014 25500 mvcc.cc:205] Tried to move back new op lower bound from 7317450781610291200 to 7317450781448445952. Current Snapshot: MvccSnapshot[applied={T|T < 7317450781610291200}]
I20260811 22:23:27.248844 24683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:27.249056 24683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:27.251801 24683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Sending DeleteTablet for 3 replicas of tablet 525e25e504374696bc860bb69eaf2380
I20260811 22:23:27.252803 25322 tablet_service.cc:1514] Processing DeleteTablet for tablet 525e25e504374696bc860bb69eaf2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:27 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:33476
I20260811 22:23:27.252828 25113 tablet_service.cc:1514] Processing DeleteTablet for tablet 525e25e504374696bc860bb69eaf2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:27 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:54886
I20260811 22:23:27.252844 24893 tablet_service.cc:1514] Processing DeleteTablet for tablet 525e25e504374696bc860bb69eaf2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:27 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:47786
I20260811 22:23:27.252975 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 24790
I20260811 22:23:27.258132 25514 tablet_replica.cc:330] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: stopping tablet replica
I20260811 22:23:27.258528 25513 tablet_replica.cc:330] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: stopping tablet replica
I20260811 22:23:27.258874 25513 raft_consensus.cc:2229] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:27.259251 25513 raft_consensus.cc:2258] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:27.259706 25514 raft_consensus.cc:2229] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:23:27.260089 25514 raft_consensus.cc:2258] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:27.267243 25514 ts_tablet_manager.cc:1905] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:27.267452 25513 ts_tablet_manager.cc:1905] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:27.270128 25514 ts_tablet_manager.cc:1918] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:27.270253 25514 log.cc:1196] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/525e25e504374696bc860bb69eaf2380
I20260811 22:23:27.270598 25513 ts_tablet_manager.cc:1918] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:27.270620 25514 ts_tablet_manager.cc:1939] T 525e25e504374696bc860bb69eaf2380 P a5d58361fb004a7cb54d23491d52c821: Deleting consensus metadata
I20260811 22:23:27.270808 25513 log.cc:1196] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/525e25e504374696bc860bb69eaf2380
I20260811 22:23:27.271158 25513 ts_tablet_manager.cc:1939] T 525e25e504374696bc860bb69eaf2380 P 532715aa46b244a2a96f3cc396e7eef3: Deleting consensus metadata
I20260811 22:23:27.271979 24657 catalog_manager.cc:4841] TS a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): tablet 525e25e504374696bc860bb69eaf2380 (table test-table [id=f3034b52c32c4887a0823d6e60e0f558]) successfully deleted
I20260811 22:23:27.272678 24655 catalog_manager.cc:4841] TS 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): tablet 525e25e504374696bc860bb69eaf2380 (table test-table [id=f3034b52c32c4887a0823d6e60e0f558]) successfully deleted
W20260811 22:23:27.281114 24660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:35483 (error 108)
I20260811 22:23:27.281301 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 25009
I20260811 22:23:27.311832 25464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:27.294105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35483 (local address 127.0.0.1:47798)
0811 22:23:27.311553 (+ 17448us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:27.311684 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17291}
W20260811 22:23:27.312155 24660 catalog_manager.cc:4584] TS 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 525e25e504374696bc860bb69eaf2380: Network error: Client connection negotiation failed: client connection to 127.29.3.65:35483: connect: Connection refused (error 111)
I20260811 22:23:27.321148 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 25226
I20260811 22:23:27.346727 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 24616
I20260811 22:23:47.367573 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38759
--webserver_interface=127.29.3.126
--webserver_port=44143
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:23:47.484135 25518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:47.484489 25518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:47.484539 25518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:47.488783 25518 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 22:23:47.488859 25518 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 22:23:47.488875 25518 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 22:23:47.488888 25518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:23:47.488905 25518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:47.488921 25518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:47.488936 25518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:23:47.488950 25518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:23:47.494524 25518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:47.495616 25518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:47.496624 25518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:47.510619 25526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:47.510777 25528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:47.510619 25530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:47.510994 25518 server_base.cc:1034] running on GCE node
I20260811 22:23:47.511711 25518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:47.512918 25518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:47.514086 25518 hybrid_clock.cc:648] HybridClock initialized: now 1786487027514064 us; error 41 us; skew 500 ppm
Aug 11 22:23:47 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487027, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:47.516896 25518 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:23:47.517831 25518 webserver.cc:460] Webserver started at http://127.29.3.126:44143/ using document root <none> and password file <none>
I20260811 22:23:47.518457 25518 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:47.518550 25518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:47.522645 25518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260811 22:23:47.524930 25546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:47.526449 25518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:47.526598 25518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "b62badc91ad247c1bf5d18c8fe4563ae"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "8a269d9d89fe21ea1b49f8b2f58b4fb6"
server_key_iv: "9ee137dda81e2c8d478bd5507aa56424"
server_key_version: "encryptionkey@0"
I20260811 22:23:47.527254 25518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:47.546782 25518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:47.549901 25518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:47.550112 25518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:47.557622 25518 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:38759
I20260811 22:23:47.557902 25622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:38759 every 8 connection(s)
I20260811 22:23:47.558900 25518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:23:47.563620 25623 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:23:47.565464 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap starting.
I20260811 22:23:47.565706 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 25518
I20260811 22:23:47.566200 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:35483
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:38759
--webserver_port=40049
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:47.585050 25623 log.cc:824] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:47.594198 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:23:47.594657 25623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap complete.
I20260811 22:23:47.598855 25623 raft_consensus.cc:348] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:47.599308 25623 raft_consensus.cc:729] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62badc91ad247c1bf5d18c8fe4563ae, State: Initialized, Role: FOLLOWER
I20260811 22:23:47.599870 25623 consensus_queue.cc:260] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:47.599995 25623 raft_consensus.cc:388] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:23:47.600070 25623 raft_consensus.cc:482] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:23:47.600172 25623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 1 FOLLOWER]: Advancing to term 2
I20260811 22:23:47.602494 25623 raft_consensus.cc:504] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:47.602988 25623 leader_election.cc:302] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b62badc91ad247c1bf5d18c8fe4563ae; no voters:
I20260811 22:23:47.603366 25623 leader_election.cc:288] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 22:23:47.603521 25627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Leader election won for term 2
I20260811 22:23:47.603807 25627 raft_consensus.cc:686] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 LEADER]: Becoming Leader. State: Replica: b62badc91ad247c1bf5d18c8fe4563ae, State: Running, Role: LEADER
I20260811 22:23:47.604159 25627 consensus_queue.cc:237] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:23:47.604641 25623 sys_catalog.cc:564] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:23:47.605291 25627 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b62badc91ad247c1bf5d18c8fe4563ae. Latest consensus state: current_term: 2 leader_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } } }
I20260811 22:23:47.605978 25627 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:47.605865 25628 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } } }
I20260811 22:23:47.606818 25628 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: This master's current role is: LEADER
I20260811 22:23:47.610040 25658 catalog_manager.cc:1259] Loaded cluster ID: a4bac1c3a8564a43bfe2fce718af424e
I20260811 22:23:47.610102 25658 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: loading cluster ID for follower catalog manager: success
I20260811 22:23:47.612700 25658 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: acquiring CA information for follower catalog manager: success
I20260811 22:23:47.614452 25658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:23:47.614652 25649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:23:47.617233 25649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:23:47.617748 25649 catalog_manager.cc:1259] Loaded cluster ID: a4bac1c3a8564a43bfe2fce718af424e
I20260811 22:23:47.617810 25649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:23:47.618611 25649 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:23:47.619095 25649 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Loaded TSK: 0
I20260811 22:23:47.633632 25649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Generated new TSK 1
I20260811 22:23:47.634061 25649 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:23:47.688568 25625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:47.688890 25625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:47.688961 25625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:47.693277 25625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:47.693356 25625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:47.693439 25625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:23:47.699097 25625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:35483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=40049
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:47.700198 25625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:47.701217 25625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:47.711264 25668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:47.711436 25669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:47.711275 25674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:47.711838 25625 server_base.cc:1034] running on GCE node
I20260811 22:23:47.712282 25625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:47.712970 25625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:47.714159 25625 hybrid_clock.cc:648] HybridClock initialized: now 1786487027714116 us; error 52 us; skew 500 ppm
Aug 11 22:23:47 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487027, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:47.716974 25625 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:23:47.717948 25625 webserver.cc:460] Webserver started at http://127.29.3.65:40049/ using document root <none> and password file <none>
I20260811 22:23:47.718572 25625 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:47.718657 25625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:47.722730 25625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:47.725045 25685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:47.726577 25625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:47.726711 25625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "8f3055bb4c254788a73c91ddc8ee5e92"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "03d59ff313ea61c5f811cd9941be2ee8"
server_key_iv: "d9421c3fec00d70c75b26ee603f3c5a1"
server_key_version: "encryptionkey@0"
I20260811 22:23:47.727317 25625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:47.743885 25625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:47.747306 25625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:47.747512 25625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:47.748160 25625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:47.749305 25697 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 22:23:47.754834 25625 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 22:23:47.754921 25625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260811 22:23:47.754994 25625 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 22:23:47.756798 25625 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 22:23:47.756855 25625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260811 22:23:47.757162 25697 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap starting.
I20260811 22:23:47.767422 25625 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:35483
I20260811 22:23:47.767521 25892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:35483 every 8 connection(s)
I20260811 22:23:47.768443 25625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:23:47.774564 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 25625
I20260811 22:23:47.775022 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:45681
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:38759
--webserver_port=39893
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:23:47.776996 25697 log.cc:824] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:23:47 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487027, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:47.785195 25697 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:23:47.785668 25697 tablet_bootstrap.cc:492] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap complete.
I20260811 22:23:47.787068 25697 ts_tablet_manager.cc:1397] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.011s
I20260811 22:23:47.788244 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:47.770482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:52803)
0811 22:23:47.771092 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:47.771162 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:47.772307 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:47.773513 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:47.773523 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:47.774028 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:47.774805 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:47.774831 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.776587 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.776591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:47.777776 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:47.777787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.778752 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.778756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:47.778960 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:47.779665 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:47.779685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:47.781563 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:47.784448 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:47.784455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:47.784468 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:47.784834 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:47.785297 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:47.785300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:47.785302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:47.785481 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:47.786284 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:47.786314 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:47.786664 (+ 350us) client_negotiation.cc:769] Sending connection context
0811 22:23:47.786865 (+ 201us) client_negotiation.cc:240] Negotiation successful
0811 22:23:47.787288 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":142,"threads_started":1}
I20260811 22:23:47.788242 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:47.770591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:52803 (local address 127.29.3.126:38759)
0811 22:23:47.771123 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:47.771145 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:47.771292 (+ 147us) server_negotiation.cc:407] Connection header received
0811 22:23:47.772772 (+ 1480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:47.772784 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:47.773091 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:47.773380 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:47.775019 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.776416 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.777901 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.778601 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.778784 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:47.781801 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:47.781809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:47.781828 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:47.781870 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:47.784306 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:47.784970 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:47.784977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:47.784984 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:47.785174 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:47.785597 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:47.785601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:47.785603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:47.786146 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:47.786347 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:47.787031 (+ 684us) server_negotiation.cc:299] Negotiation successful
0811 22:23:47.787390 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":149,"threads_started":1}
I20260811 22:23:47.790025 25697 raft_consensus.cc:348] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:47.790529 25697 raft_consensus.cc:729] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3055bb4c254788a73c91ddc8ee5e92, State: Initialized, Role: FOLLOWER
I20260811 22:23:47.791016 25697 consensus_queue.cc:260] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } } peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } }
I20260811 22:23:47.791503 25893 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38759
I20260811 22:23:47.791747 25893 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:47.792037 25697 ts_tablet_manager.cc:1428] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260811 22:23:47.792320 25893 heartbeater.cc:511] Master 127.29.3.126:38759 requested a full tablet report, sending...
I20260811 22:23:47.795293 25572 ts_manager.cc:194] Registered new tserver with Master: 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483)
I20260811 22:23:47.799614 25572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:52803
I20260811 22:23:47.802681 25893 heartbeater.cc:503] Master 127.29.3.126:38759 was elected leader, sending a full tablet report...
Aug 11 22:23:47 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487027, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.65@KRBTEST.COM
I20260811 22:23:47.809844 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:47.798912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35483 (local address 127.0.0.1:60654)
0811 22:23:47.799572 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:47.799626 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:47.799798 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:47.800386 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:47.800392 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:47.800872 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:47.800990 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:47.800998 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.802142 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.802146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:47.802921 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:47.802933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.803568 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.803572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:47.803690 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:47.804233 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:47.804251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:47.806038 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:47.808025 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:47.808037 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:47.808044 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:47.808298 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:47.808596 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:47.808599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:47.808601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:47.808684 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:47.809151 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:47.809158 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:47.809296 (+ 138us) client_negotiation.cc:769] Sending connection context
0811 22:23:47.809418 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 22:23:47.809653 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":117,"threads_started":1}
I20260811 22:23:47.809844 25910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:47.799020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.29.3.65:35483)
0811 22:23:47.799910 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:47.799923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:47.799948 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:23:47.800013 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:47.800017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:47.800230 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:47.800366 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:47.801115 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.802002 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.803055 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.803455 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.803552 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:47.806181 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:47.806191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:47.806203 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:47.806229 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:47.807911 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:47.808412 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:47.808419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:47.808421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:47.808508 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:47.808767 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:47.808771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:47.808773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:47.809039 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:47.809161 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:47.809516 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 22:23:47.809684 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":116,"threads_started":1}
I20260811 22:23:47.811035 25767 tablet_service.cc:1514] Processing DeleteTablet for tablet 525e25e504374696bc860bb69eaf2380 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:27 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:60654
I20260811 22:23:47.812103 25911 tablet_replica.cc:330] stopping tablet replica
I20260811 22:23:47.812415 25911 raft_consensus.cc:2229] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:47.812659 25911 raft_consensus.cc:2258] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:47.813620 25911 ts_tablet_manager.cc:1905] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:47.817322 25911 ts_tablet_manager.cc:1918] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:47.817418 25911 log.cc:1196] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal/wals/525e25e504374696bc860bb69eaf2380
I20260811 22:23:47.817647 25911 ts_tablet_manager.cc:1939] T 525e25e504374696bc860bb69eaf2380 P 8f3055bb4c254788a73c91ddc8ee5e92: Deleting consensus metadata
I20260811 22:23:47.819464 25552 catalog_manager.cc:4841] TS 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483): tablet 525e25e504374696bc860bb69eaf2380 (table test-table [id=f3034b52c32c4887a0823d6e60e0f558]) successfully deleted
W20260811 22:23:47.901156 25901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:47.901458 25901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:47.901504 25901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:47.905786 25901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:47.905860 25901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:47.905936 25901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:23:47.911675 25901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:45681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=39893
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:47.912833 25901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:47.913852 25901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:47.924003 25921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:47.923990 25917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:47.924069 25901 server_base.cc:1034] running on GCE node
W20260811 22:23:47.923990 25918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:47.924700 25901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:47.925390 25901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:47.926565 25901 hybrid_clock.cc:648] HybridClock initialized: now 1786487027926526 us; error 51 us; skew 500 ppm
Aug 11 22:23:47 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487027, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:47.929550 25901 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:23:47.930562 25901 webserver.cc:460] Webserver started at http://127.29.3.66:39893/ using document root <none> and password file <none>
I20260811 22:23:47.931223 25901 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:47.931310 25901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:47.935619 25901 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:47.937796 25933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:47.939337 25901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:23:47.939453 25901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "532715aa46b244a2a96f3cc396e7eef3"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "4e007db31875c9d8f94556ee8d65ada9"
server_key_iv: "70a275fe8432f4dc9cd5fe918e4b4e4f"
server_key_version: "encryptionkey@0"
I20260811 22:23:47.940040 25901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:47.976262 25901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:47.979597 25901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:47.979794 25901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:47.980417 25901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:47.981295 25901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:47.981345 25901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:47.981398 25901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:47.981426 25901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:47.990105 25901 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:45681
I20260811 22:23:47.990139 26116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:45681 every 8 connection(s)
I20260811 22:23:47.991158 25901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:23:47.993338 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 25901
I20260811 22:23:47.993831 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:35367
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:38759
--webserver_port=37463
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 22:23:47 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487027, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:48.006408 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:47.992934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:36917 (local address 127.29.3.126:38759)
0811 22:23:47.993070 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:47.993073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:47.993992 (+ 919us) server_negotiation.cc:407] Connection header received
0811 22:23:47.995716 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:47.995720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:47.995793 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:47.995878 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:47.997086 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.997835 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.999240 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.999603 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.999733 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.002350 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:48.002354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:48.002357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:48.002396 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:48.004091 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.004641 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.004645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.004647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.004746 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.005106 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.005110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.005112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.005400 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:48.005509 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.006045 (+ 536us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.006203 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:23:48.007304 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:47.993298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:36917)
0811 22:23:47.993815 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:47.993875 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:47.995430 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:47.996044 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:47.996051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:47.996445 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:47.996933 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:47.996947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.997973 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.997976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:47.999108 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:47.999119 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:47.999807 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:47.999811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:47.999963 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.000639 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:48.000657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:48.002127 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:48.004232 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:48.004237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:48.004251 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:48.004509 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:48.004833 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:48.004835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:48.004836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:48.004985 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:48.005504 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:48.005510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:48.005795 (+ 285us) client_negotiation.cc:769] Sending connection context
0811 22:23:48.005981 (+ 186us) client_negotiation.cc:240] Negotiation successful
0811 22:23:48.006374 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260811 22:23:48.008497 26117 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38759
I20260811 22:23:48.008788 26117 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:48.009383 26117 heartbeater.cc:511] Master 127.29.3.126:38759 requested a full tablet report, sending...
I20260811 22:23:48.010597 25572 ts_manager.cc:194] Registered new tserver with Master: 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681)
I20260811 22:23:48.011219 25572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:36917
W20260811 22:23:48.113024 26122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:23:48.113332 26122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:23:48.113397 26122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:23:48.117591 26122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:23:48.117672 26122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:23:48.117754 26122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:23:48.123334 26122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:35367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=37463
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:23:48.124778 26122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:23:48.126174 26122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:23:48.141433 26135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:23:48.141441 26131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:48.141642 26122 server_base.cc:1034] running on GCE node
W20260811 22:23:48.141525 26132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:23:48.142138 26122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:23:48.142858 26122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:23:48.144018 26122 hybrid_clock.cc:648] HybridClock initialized: now 1786487028143994 us; error 41 us; skew 500 ppm
Aug 11 22:23:48 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487028, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:23:48.146832 26122 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:23:48.147727 26122 webserver.cc:460] Webserver started at http://127.29.3.67:37463/ using document root <none> and password file <none>
I20260811 22:23:48.148280 26122 fs_manager.cc:359] Metadata directory not provided
I20260811 22:23:48.148355 26122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:23:48.152443 26122 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:23:48.154598 26151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:48.155875 26122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:23:48.155990 26122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "a5d58361fb004a7cb54d23491d52c821"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "09a350fa4725f962d72608ba2ba93962"
server_key_iv: "5eb9f2e38a5cc53e85200356efdd260e"
server_key_version: "encryptionkey@0"
I20260811 22:23:48.156544 26122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:23:48.181871 26122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:23:48.185274 26122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:23:48.185456 26122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:23:48.186087 26122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:23:48.187005 26122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:23:48.187057 26122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:48.187111 26122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:23:48.187137 26122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:23:48.196133 26122 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:35367
I20260811 22:23:48.196182 26332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:35367 every 8 connection(s)
I20260811 22:23:48.197161 26122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:23:48.206060 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 26122
Aug 11 22:23:48 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487028, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:23:48.211410 25900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.199042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:33041 (local address 127.29.3.126:38759)
0811 22:23:48.199184 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.199188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.200067 (+ 879us) server_negotiation.cc:407] Connection header received
0811 22:23:48.201398 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.201402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.201465 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.201540 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:48.202841 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.203370 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.204596 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.204841 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.204933 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.207801 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:48.207805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:48.207809 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:48.207836 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:48.209229 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.209724 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.209728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.209730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.209818 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.210140 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.210143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.210145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.210382 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:48.210474 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.211089 (+ 615us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.211242 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:23:48.212380 26336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.199436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:33041)
0811 22:23:48.199905 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:48.199950 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:48.201160 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:48.201754 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:48.201762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:48.202149 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:23:48.202677 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.202692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.203499 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.203502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.204427 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.204437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.205000 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.205003 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.205155 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.205870 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:23:48.205887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:23:48.207633 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:23:48.209337 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:48.209342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:48.209356 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:48.209617 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:48.209902 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:23:48.209904 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:23:48.209906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:23:48.210039 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:23:48.210485 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:23:48.210492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:23:48.210809 (+ 317us) client_negotiation.cc:769] Sending connection context
0811 22:23:48.211032 (+ 223us) client_negotiation.cc:240] Negotiation successful
0811 22:23:48.211394 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
I20260811 22:23:48.213690 26333 heartbeater.cc:347] Connected to a master server at 127.29.3.126:38759
I20260811 22:23:48.213979 26333 heartbeater.cc:464] Registering TS with master...
I20260811 22:23:48.214676 26333 heartbeater.cc:511] Master 127.29.3.126:38759 requested a full tablet report, sending...
I20260811 22:23:48.214766 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.206811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.29.3.126:38759)
0811 22:23:48.207403 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.207408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.207419 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:23:48.207450 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.207452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.207488 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.207564 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:48.208439 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.209073 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.210068 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.210342 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.210432 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.211693 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:48.211696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:48.211699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:48.211724 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:48.213063 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.213483 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.213487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.213488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.213576 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.213859 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.213862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.213864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.214084 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:48.214245 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.214476 (+ 231us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.214592 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":88,"threads_started":1}
I20260811 22:23:48.215793 25572 ts_manager.cc:194] Registered new tserver with Master: a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
I20260811 22:23:48.216476 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:23:48.216501 25572 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:33041
I20260811 22:23:48.220706 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.217279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.29.3.126:38759)
0811 22:23:48.217383 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.217386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.217575 (+ 189us) server_negotiation.cc:407] Connection header received
0811 22:23:48.217758 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.217761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.217858 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.217944 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:23:48.218182 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.218744 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.219575 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.219813 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.219900 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.220202 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:23:48.220335 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 22:23:48.220580 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34554: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260811 22:23:48.220772 26340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34554: token expired
W20260811 22:23:48.221086 25449 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.3.126:38759)
I20260811 22:23:48.227375 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.221416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.29.3.126:38759)
0811 22:23:48.221514 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.221517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.221531 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:23:48.221621 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.221623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.221658 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.221721 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:23:48.222259 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.222754 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.223481 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.223672 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.223736 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.224864 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:23:48.224866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:23:48.224868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:23:48.224890 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:23:48.226185 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.226527 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.226530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.226531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.226603 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:23:48.226777 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:23:48.226779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:23:48.226780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:23:48.226927 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:23:48.227018 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.227177 (+ 159us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.227267 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 22:23:48.229653 25572 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:23:48.233110 25572 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:23:48.241009 25767 tablet_service.cc:1467] Processing CreateTablet for tablet b8a92cd691e34c9ebad7a0fe15932a9b (DEFAULT_TABLE table=test-table [id=508c9f9d71cc447185787212e684ec33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:48.241760 25767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a92cd691e34c9ebad7a0fe15932a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:48.244875 25697 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap starting.
I20260811 22:23:48.245065 25909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.239560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45681 (local address 127.0.0.1:44400)
0811 22:23:48.239661 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:48.239676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:48.239766 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:48.240690 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:48.240694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:48.240714 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:48.240821 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.240826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.242340 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.242343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.243691 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.243704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.244591 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.244594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.244706 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.244756 (+ 50us) client_negotiation.cc:769] Sending connection context
0811 22:23:48.244831 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 22:23:48.244925 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":49}
I20260811 22:23:48.245397 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.239745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.29.3.66:45681)
0811 22:23:48.240123 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.240136 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.240163 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:23:48.240262 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.240267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.240512 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.240955 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:48.241105 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.242223 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.243818 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.244471 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.244573 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.244749 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.244863 (+ 114us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.245059 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":170,"threads_started":1}
I20260811 22:23:48.245437 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.239839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:35367 (local address 127.0.0.1:56888)
0811 22:23:48.240298 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:48.240311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:48.240395 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:48.241198 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:48.241201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:48.241220 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:48.241323 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.241329 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.242663 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.242667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.243835 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.243847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.245032 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.245035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.245136 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.245162 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 22:23:48.245212 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 22:23:48.245295 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":9,"thread_start_us":88,"threads_started":1}
I20260811 22:23:48.245869 25697 tablet_bootstrap.cc:654] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:48.246140 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.240264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.29.3.67:35367)
0811 22:23:48.240644 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.240672 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.240766 (+ 94us) server_negotiation.cc:407] Connection header received
0811 22:23:48.240844 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.240849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.241039 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.241200 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:48.241434 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.242546 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.243961 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.244898 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.245713 (+ 815us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.245861 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.245912 (+ 51us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.246001 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":173,"threads_started":1}
I20260811 22:23:48.248564 26229 tablet_service.cc:1467] Processing CreateTablet for tablet b8a92cd691e34c9ebad7a0fe15932a9b (DEFAULT_TABLE table=test-table [id=508c9f9d71cc447185787212e684ec33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:48.248559 26009 tablet_service.cc:1467] Processing CreateTablet for tablet b8a92cd691e34c9ebad7a0fe15932a9b (DEFAULT_TABLE table=test-table [id=508c9f9d71cc447185787212e684ec33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:23:48.249583 26009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a92cd691e34c9ebad7a0fe15932a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:48.249599 26229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a92cd691e34c9ebad7a0fe15932a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:23:48.250250 25697 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: No bootstrap required, opened a new log
I20260811 22:23:48.250388 25697 ts_tablet_manager.cc:1397] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260811 22:23:48.250914 25697 raft_consensus.cc:348] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.251052 25697 raft_consensus.cc:374] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:48.251096 25697 raft_consensus.cc:729] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3055bb4c254788a73c91ddc8ee5e92, State: Initialized, Role: FOLLOWER
I20260811 22:23:48.251233 25697 consensus_queue.cc:260] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.251506 25697 ts_tablet_manager.cc:1428] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 22:23:48.257377 26361 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Bootstrap starting.
I20260811 22:23:48.257377 26364 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Bootstrap starting.
I20260811 22:23:48.259462 26361 tablet_bootstrap.cc:654] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:48.259785 26364 tablet_bootstrap.cc:654] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Neither blocks nor log segments found. Creating new log.
I20260811 22:23:48.260592 26361 log.cc:824] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:48.260592 26364 log.cc:824] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Log is configured to *not* fsync() on all Append() calls
I20260811 22:23:48.262346 26364 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: No bootstrap required, opened a new log
I20260811 22:23:48.262369 26361 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: No bootstrap required, opened a new log
I20260811 22:23:48.262530 26361 ts_tablet_manager.cc:1397] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:23:48.262531 26364 ts_tablet_manager.cc:1397] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:23:48.265525 26364 raft_consensus.cc:348] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.265524 26361 raft_consensus.cc:348] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.265750 26364 raft_consensus.cc:374] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:48.265753 26361 raft_consensus.cc:374] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:23:48.265787 26364 raft_consensus.cc:729] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 532715aa46b244a2a96f3cc396e7eef3, State: Initialized, Role: FOLLOWER
I20260811 22:23:48.265813 26361 raft_consensus.cc:729] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d58361fb004a7cb54d23491d52c821, State: Initialized, Role: FOLLOWER
I20260811 22:23:48.266202 26364 consensus_queue.cc:260] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.266201 26361 consensus_queue.cc:260] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.266870 26333 heartbeater.cc:503] Master 127.29.3.126:38759 was elected leader, sending a full tablet report...
I20260811 22:23:48.267114 26117 heartbeater.cc:503] Master 127.29.3.126:38759 was elected leader, sending a full tablet report...
I20260811 22:23:48.267163 26364 ts_tablet_manager.cc:1428] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260811 22:23:48.268004 26361 ts_tablet_manager.cc:1428] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260811 22:23:48.269687 25897 tablet.cc:2369] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:23:48.301234 25908 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:23:48.301505 25908 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.302765 25908 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
I20260811 22:23:48.306893 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.303105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:50485 (local address 127.29.3.66:45681)
0811 22:23:48.303259 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.303264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.303461 (+ 197us) server_negotiation.cc:407] Connection header received
0811 22:23:48.303628 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.303631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.303678 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.303751 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:48.304043 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.304537 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.305815 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.306258 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.306374 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.306439 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.306687 (+ 248us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.306773 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:23:48.306895 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.303167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:57601 (local address 127.29.3.67:35367)
0811 22:23:48.303298 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.303302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.303510 (+ 208us) server_negotiation.cc:407] Connection header received
0811 22:23:48.303630 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.303632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.303686 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.303747 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:23:48.304042 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.304552 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.305792 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.306237 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.306360 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.306428 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.306694 (+ 266us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.306772 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:23:48.306916 26371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.302983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45681 (local address 127.29.3.65:50485)
0811 22:23:48.303365 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:48.303377 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:48.303502 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:48.303768 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:48.303770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:48.303826 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:48.303940 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.303945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.304636 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.304640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.305683 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.305694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.306410 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.306413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.306535 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.306554 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 22:23:48.306618 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:23:48.306719 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":96,"spinlock_wait_cycles":58368,"thread_start_us":121,"threads_started":1}
I20260811 22:23:48.306988 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.303049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:35367 (local address 127.29.3.65:57601)
0811 22:23:48.303411 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:48.303423 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:23:48.303527 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:23:48.303766 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:23:48.303769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:23:48.303789 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:23:48.303939 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.303945 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.304658 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.304661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.305671 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:23:48.305682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.306388 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.306392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:23:48.306488 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.306553 (+ 65us) client_negotiation.cc:769] Sending connection context
0811 22:23:48.306619 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 22:23:48.306739 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":96,"thread_start_us":86,"threads_started":1}
I20260811 22:23:48.307559 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a92cd691e34c9ebad7a0fe15932a9b" candidate_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532715aa46b244a2a96f3cc396e7eef3" is_pre_election: true
I20260811 22:23:48.307708 26263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a92cd691e34c9ebad7a0fe15932a9b" candidate_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d58361fb004a7cb54d23491d52c821" is_pre_election: true
I20260811 22:23:48.307888 26042 raft_consensus.cc:2454] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3055bb4c254788a73c91ddc8ee5e92 in term 0.
I20260811 22:23:48.307998 26263 raft_consensus.cc:2454] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3055bb4c254788a73c91ddc8ee5e92 in term 0.
I20260811 22:23:48.308586 25691 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92, a5d58361fb004a7cb54d23491d52c821; no voters:
I20260811 22:23:48.308890 25908 raft_consensus.cc:2781] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:23:48.308977 25908 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:23:48.309026 25908 raft_consensus.cc:3037] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:48.310124 25908 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.310519 25908 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 1 election: Requested vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
I20260811 22:23:48.310899 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a92cd691e34c9ebad7a0fe15932a9b" candidate_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "532715aa46b244a2a96f3cc396e7eef3"
I20260811 22:23:48.311028 26042 raft_consensus.cc:3037] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:48.312098 26042 raft_consensus.cc:2454] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3055bb4c254788a73c91ddc8ee5e92 in term 1.
I20260811 22:23:48.312399 26263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a92cd691e34c9ebad7a0fe15932a9b" candidate_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5d58361fb004a7cb54d23491d52c821"
I20260811 22:23:48.312446 25689 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 532715aa46b244a2a96f3cc396e7eef3, 8f3055bb4c254788a73c91ddc8ee5e92; no voters:
I20260811 22:23:48.312526 26263 raft_consensus.cc:3037] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:23:48.312613 25908 raft_consensus.cc:2781] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:23:48.312834 25908 raft_consensus.cc:686] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 LEADER]: Becoming Leader. State: Replica: 8f3055bb4c254788a73c91ddc8ee5e92, State: Running, Role: LEADER
I20260811 22:23:48.313313 25908 consensus_queue.cc:237] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:23:48.313839 26263 raft_consensus.cc:2454] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3055bb4c254788a73c91ddc8ee5e92 in term 1.
I20260811 22:23:48.316579 25572 catalog_manager.cc:5495] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } health_report { overall_health: HEALTHY } } }
I20260811 22:23:48.333630 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.329758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.29.3.65:35483)
0811 22:23:48.330023 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0811 22:23:48.330027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:23:48.330039 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:23:48.330116 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:23:48.330118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:23:48.330184 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:23:48.330335 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:23:48.330585 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.331167 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.332191 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:23:48.332426 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:23:48.332513 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:23:48.332736 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:23:48.333208 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:23:48.333294 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:23:48.333409 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 22:23:48.333474 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260811 22:23:48.337599 26263 raft_consensus.cc:1261] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Refusing update from remote peer 8f3055bb4c254788a73c91ddc8ee5e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:48.337598 26042 raft_consensus.cc:1261] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Refusing update from remote peer 8f3055bb4c254788a73c91ddc8ee5e92: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:23:48.338341 25908 consensus_queue.cc:1035] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:48.338524 26373 consensus_queue.cc:1035] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:23:48.346958 26385 mvcc.cc:205] Tried to move back new op lower bound from 7317450868064763904 to 7317450867974725632. Current Snapshot: MvccSnapshot[applied={T|T < 7317450868064763904}]
I20260811 22:23:48.354575 25571 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:48.354744 25571 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:23:48.357064 25571 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Sending DeleteTablet for 3 replicas of tablet b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:23:48.357824 26009 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a92cd691e34c9ebad7a0fe15932a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:48 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44400
I20260811 22:23:48.357882 26229 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a92cd691e34c9ebad7a0fe15932a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:48 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:56888
I20260811 22:23:48.358022 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 25625
I20260811 22:23:48.358103 25767 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a92cd691e34c9ebad7a0fe15932a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:23:48 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:60654
I20260811 22:23:48.363289 26400 tablet_replica.cc:330] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: stopping tablet replica
I20260811 22:23:48.363749 26400 raft_consensus.cc:2229] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:48.364208 26400 raft_consensus.cc:2258] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:48.369679 26396 tablet_replica.cc:330] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: stopping tablet replica
I20260811 22:23:48.370023 26396 raft_consensus.cc:2229] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:23:48.370434 26396 raft_consensus.cc:2258] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:23:48.370445 26400 ts_tablet_manager.cc:1905] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:48.371865 26396 ts_tablet_manager.cc:1905] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:23:48.373319 26400 ts_tablet_manager.cc:1918] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:48.373426 26400 log.cc:1196] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:23:48.373797 26400 ts_tablet_manager.cc:1939] T b8a92cd691e34c9ebad7a0fe15932a9b P a5d58361fb004a7cb54d23491d52c821: Deleting consensus metadata
I20260811 22:23:48.374081 26396 ts_tablet_manager.cc:1918] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:23:48.374176 26396 log.cc:1196] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:23:48.374495 26396 ts_tablet_manager.cc:1939] T b8a92cd691e34c9ebad7a0fe15932a9b P 532715aa46b244a2a96f3cc396e7eef3: Deleting consensus metadata
I20260811 22:23:48.375329 25549 catalog_manager.cc:4841] TS a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): tablet b8a92cd691e34c9ebad7a0fe15932a9b (table test-table [id=508c9f9d71cc447185787212e684ec33]) successfully deleted
I20260811 22:23:48.375582 25548 catalog_manager.cc:4841] TS 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): tablet b8a92cd691e34c9ebad7a0fe15932a9b (table test-table [id=508c9f9d71cc447185787212e684ec33]) successfully deleted
W20260811 22:23:48.376657 25552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:35483 (error 108)
I20260811 22:23:48.377463 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 25901
I20260811 22:23:48.377693 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:23:48.377406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:35483 (local address 127.0.0.1:60672)
0811 22:23:48.377540 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 22:23:48.377609 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:35483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 22:23:48.391743 25552 catalog_manager.cc:4584] TS 8f3055bb4c254788a73c91ddc8ee5e92 (127.29.3.65:35483): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8a92cd691e34c9ebad7a0fe15932a9b: Network error: Client connection negotiation failed: client connection to 127.29.3.65:35483: connect: Connection refused (error 111)
I20260811 22:23:48.402168 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 26122
I20260811 22:23:48.415809 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 25518
I20260811 22:24:08.434152 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38759
--webserver_interface=127.29.3.126
--webserver_port=44143
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:38759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:08.553902 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:08.554190 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:08.554231 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:08.558396 26404 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 22:24:08.558465 26404 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 22:24:08.558481 26404 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 22:24:08.558493 26404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:08.558514 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:08.558547 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:08.558574 26404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:08.558599 26404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:08.564105 26404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:38759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:38759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=44143
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:08.565277 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:08.566372 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:08.575028 26417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:08.575040 26422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:08.575083 26416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:08.575407 26404 server_base.cc:1034] running on GCE node
I20260811 22:24:08.576061 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:08.577284 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:08.578451 26404 hybrid_clock.cc:648] HybridClock initialized: now 1786487048578433 us; error 32 us; skew 500 ppm
Aug 11 22:24:08 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487048, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:08.581457 26404 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:08.582506 26404 webserver.cc:460] Webserver started at http://127.29.3.126:44143/ using document root <none> and password file <none>
I20260811 22:24:08.583068 26404 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:08.583148 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:08.587230 26404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 22:24:08.589404 26430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:08.590714 26404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:24:08.590839 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "b62badc91ad247c1bf5d18c8fe4563ae"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "8a269d9d89fe21ea1b49f8b2f58b4fb6"
server_key_iv: "9ee137dda81e2c8d478bd5507aa56424"
server_key_version: "encryptionkey@0"
I20260811 22:24:08.591414 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:08.606819 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:08.609965 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:08.610172 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:08.617667 26404 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:38759
I20260811 22:24:08.617942 26537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:38759 every 8 connection(s)
I20260811 22:24:08.618883 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:08.622506 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 26404
I20260811 22:24:08.623027 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:35483
--local_ip_for_outbound_sockets=127.29.3.65
--tserver_master_addrs=127.29.3.126:38759
--webserver_port=40049
--webserver_interface=127.29.3.65
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:24:08.623847 26538 sys_catalog.cc:263] Verifying existing consensus state
I20260811 22:24:08.625970 26538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap starting.
I20260811 22:24:08.645008 26538 log.cc:824] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:08.656721 26538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:24:08.657137 26538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: Bootstrap complete.
I20260811 22:24:08.661230 26538 raft_consensus.cc:348] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:24:08.661641 26538 raft_consensus.cc:729] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b62badc91ad247c1bf5d18c8fe4563ae, State: Initialized, Role: FOLLOWER
I20260811 22:24:08.662160 26538 consensus_queue.cc:260] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:24:08.662326 26538 raft_consensus.cc:388] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:08.662401 26538 raft_consensus.cc:482] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:08.662499 26538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 2 FOLLOWER]: Advancing to term 3
I20260811 22:24:08.664485 26538 raft_consensus.cc:504] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:24:08.664834 26538 leader_election.cc:302] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b62badc91ad247c1bf5d18c8fe4563ae; no voters:
I20260811 22:24:08.665164 26538 leader_election.cc:288] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 22:24:08.665364 26546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 3 FOLLOWER]: Leader election won for term 3
I20260811 22:24:08.665684 26546 raft_consensus.cc:686] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [term 3 LEADER]: Becoming Leader. State: Replica: b62badc91ad247c1bf5d18c8fe4563ae, State: Running, Role: LEADER
I20260811 22:24:08.666074 26546 consensus_queue.cc:237] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } }
I20260811 22:24:08.666456 26538 sys_catalog.cc:564] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:08.667645 26548 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b62badc91ad247c1bf5d18c8fe4563ae. Latest consensus state: current_term: 3 leader_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } } }
I20260811 22:24:08.667778 26548 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:08.668133 26547 sys_catalog.cc:455] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b62badc91ad247c1bf5d18c8fe4563ae" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 38759 } } }
I20260811 22:24:08.668232 26547 sys_catalog.cc:458] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:08.671886 26574 catalog_manager.cc:1259] Loaded cluster ID: a4bac1c3a8564a43bfe2fce718af424e
I20260811 22:24:08.671991 26574 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: loading cluster ID for follower catalog manager: success
I20260811 22:24:08.674688 26574 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b62badc91ad247c1bf5d18c8fe4563ae: acquiring CA information for follower catalog manager: success
F20260811 22:24:08.676010 26574 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fc93e6219ad google::LogMessage::Fail() at ??:0
@ 0x7fc93e62488a google::LogMessage::SendToLog() at ??:0
@ 0x7fc93e6214de google::LogMessage::Flush() at ??:0
@ 0x7fc93e622a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fc9412e518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fc948f86989 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fc948f63023 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fc948f6354b kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fc948f5c652 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fc948f5b9e9 _ZZN4kudu6master21CatalogManagerBgTasks4InitEvENKUlvE_clEv at ??:0
@ 0x7fc948fa2541 _ZNSt17_Function_handlerIFvvEZN4kudu6master21CatalogManagerBgTasks4InitEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fc940864512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fc941ea86db start_thread at ??:0
@ 0x7fc93d11f61f clone at ??:0
W20260811 22:24:08.769215 26543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:08.769629 26543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:08.769706 26543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:08.775841 26543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:08.775955 26543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:08.776050 26543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:08.784248 26543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:35483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=40049
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:08.785828 26543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:08.787315 26543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:08.799041 26623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:08.799041 26618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:08.799288 26620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:08.799289 26543 server_base.cc:1034] running on GCE node
I20260811 22:24:08.799918 26543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:08.800639 26543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:08.801802 26543 hybrid_clock.cc:648] HybridClock initialized: now 1786487048801768 us; error 52 us; skew 500 ppm
Aug 11 22:24:08 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487048, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:08.804572 26543 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:24:08.805517 26543 webserver.cc:460] Webserver started at http://127.29.3.65:40049/ using document root <none> and password file <none>
I20260811 22:24:08.806092 26543 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:08.806154 26543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:08.810374 26543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:08.812733 26632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:08.814121 26543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:08.814250 26543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "8f3055bb4c254788a73c91ddc8ee5e92"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "03d59ff313ea61c5f811cd9941be2ee8"
server_key_iv: "d9421c3fec00d70c75b26ee603f3c5a1"
server_key_version: "encryptionkey@0"
I20260811 22:24:08.814854 26543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:08.830046 26543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:08.833344 26543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:08.833530 26543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:08.834133 26543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:08.835369 26649 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 22:24:08.840077 26543 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 22:24:08.840142 26543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260811 22:24:08.840215 26543 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 22:24:08.841925 26543 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 22:24:08.841987 26543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260811 22:24:08.842324 26649 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap starting.
I20260811 22:24:08.853055 26543 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:35483
I20260811 22:24:08.853124 26829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:35483 every 8 connection(s)
I20260811 22:24:08.854104 26543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:08.861644 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 26543
I20260811 22:24:08.862161 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:45681
--local_ip_for_outbound_sockets=127.29.3.66
--tserver_master_addrs=127.29.3.126:38759
--webserver_port=39893
--webserver_interface=127.29.3.66
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:24:08.864790 26649 log.cc:824] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:08.875475 26649 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 22:24:08.876027 26649 tablet_bootstrap.cc:492] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Bootstrap complete.
I20260811 22:24:08.877657 26649 ts_tablet_manager.cc:1397] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.016s
I20260811 22:24:08.881093 26649 raft_consensus.cc:348] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:08.881687 26649 raft_consensus.cc:729] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3055bb4c254788a73c91ddc8ee5e92, State: Initialized, Role: FOLLOWER
I20260811 22:24:08.882325 26649 consensus_queue.cc:260] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:08.883510 26649 ts_tablet_manager.cc:1428] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260811 22:24:08.916507 26837 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:24:08.856692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:37731)
0811 22:24:08.857392 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:08.857463 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:08.858641 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:08.915349 (+ 56708us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":211,"threads_started":1}
W20260811 22:24:08.917130 26832 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:38759 (0 consecutive failures): Network error: Failed to ping master at 127.29.3.126:38759: Client connection negotiation failed: client connection to 127.29.3.126:38759: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 22:24:08.918010 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:08.917714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:57341)
0811 22:24:08.917854 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:08.917937 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 22:24:08.918677 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:08.918458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:51019)
0811 22:24:08.918568 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:08.918620 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 22:24:08.918835 26832 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 22:24:08.989466 26838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:08.989759 26838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:08.989804 26838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:08.993899 26838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:08.993966 26838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:08.994025 26838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:08.999370 26838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:45681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=39893
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:09.000468 26838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:09.001482 26838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:09.011404 26852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:09.011404 26854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:09.011508 26850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:09.011752 26838 server_base.cc:1034] running on GCE node
I20260811 22:24:09.012167 26838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:09.012835 26838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:09.013994 26838 hybrid_clock.cc:648] HybridClock initialized: now 1786487049013964 us; error 46 us; skew 500 ppm
Aug 11 22:24:09 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:09.016692 26838 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:24:09.017624 26838 webserver.cc:460] Webserver started at http://127.29.3.66:39893/ using document root <none> and password file <none>
I20260811 22:24:09.018213 26838 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:09.018285 26838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:09.022737 26838 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260811 22:24:09.025424 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:09.027047 26838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:24:09.027170 26838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "532715aa46b244a2a96f3cc396e7eef3"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "4e007db31875c9d8f94556ee8d65ada9"
server_key_iv: "70a275fe8432f4dc9cd5fe918e4b4e4f"
server_key_version: "encryptionkey@0"
I20260811 22:24:09.027932 26838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:09.057634 26838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:09.060936 26838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:09.061115 26838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:09.061724 26838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:09.062716 26838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:09.062767 26838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:09.062809 26838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:09.062830 26838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:09.071841 26838 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:45681
I20260811 22:24:09.071866 27059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:45681 every 8 connection(s)
I20260811 22:24:09.072954 26838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:09.076968 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.075188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:44227)
0811 22:24:09.075667 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.076178 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":120,"threads_started":1}
W20260811 22:24:09.077450 27064 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:38759 (0 consecutive failures): Network error: Failed to ping master at 127.29.3.126:38759: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
I20260811 22:24:09.078094 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.077869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:60159)
0811 22:24:09.077980 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.078033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 22:24:09.078691 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.078503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:39029)
0811 22:24:09.078598 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.078641 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 22:24:09.078866 27064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:24:09.082562 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 26838
I20260811 22:24:09.082976 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:35367
--local_ip_for_outbound_sockets=127.29.3.67
--tserver_master_addrs=127.29.3.126:38759
--webserver_port=37463
--webserver_interface=127.29.3.67
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:09.200428 27069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:09.200728 27069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:09.200795 27069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:09.204922 27069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:09.205000 27069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:09.205085 27069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:09.210899 27069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:35367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=37463
--tserver_master_addrs=127.29.3.126:38759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:09.212008 27069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:09.213044 27069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:09.223121 27076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:09.223201 27069 server_base.cc:1034] running on GCE node
W20260811 22:24:09.223114 27075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:09.223117 27078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:09.223841 27069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:09.224574 27069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:09.225751 27069 hybrid_clock.cc:648] HybridClock initialized: now 1786487049225726 us; error 38 us; skew 500 ppm
Aug 11 22:24:09 dist-test-slave-4tdj krb5kdc[24594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487049, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:09.228667 27069 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:24:09.229601 27069 webserver.cc:460] Webserver started at http://127.29.3.67:37463/ using document root <none> and password file <none>
I20260811 22:24:09.230162 27069 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:09.230226 27069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:09.234508 27069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:24:09.236686 27090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:09.238018 27069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:24:09.238152 27069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "a5d58361fb004a7cb54d23491d52c821"
format_stamp: "Formatted at 2026-08-11 22:23:26 on dist-test-slave-4tdj"
server_key: "09a350fa4725f962d72608ba2ba93962"
server_key_iv: "5eb9f2e38a5cc53e85200356efdd260e"
server_key_version: "encryptionkey@0"
I20260811 22:24:09.238793 27069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:09.267516 27069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:09.271023 27069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:09.271231 27069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:09.271869 27069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:09.272768 27069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:09.272819 27069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:09.272876 27069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:09.272912 27069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:09.281859 27069 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:35367
I20260811 22:24:09.281885 27278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:35367 every 8 connection(s)
I20260811 22:24:09.282963 27069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:09.286975 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.285191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:59135)
0811 22:24:09.285684 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.286191 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
W20260811 22:24:09.287467 27279 heartbeater.cc:644] Failed to heartbeat to 127.29.3.126:38759 (0 consecutive failures): Network error: Failed to ping master at 127.29.3.126:38759: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
I20260811 22:24:09.288115 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.287892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:41761)
0811 22:24:09.287994 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.288046 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 22:24:09.288717 27281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.288521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:45671)
0811 22:24:09.288622 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.288666 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:24:09.288869 27279 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 22:24:09.291369 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 27069
I20260811 22:24:09.919767 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:09.919358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:55369)
0811 22:24:09.919605 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:09.919666 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260811 22:24:10.080047 27286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.079521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:36083)
0811 22:24:10.079818 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:10.079891 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260811 22:24:10.289958 27287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.289524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:42395)
0811 22:24:10.289781 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:10.289841 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
I20260811 22:24:10.332276 27290 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:10.332476 27290 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:10.333719 27290 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
Aug 11 22:24:10 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487048, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.66@KRBTEST.COM
Aug 11 22:24:10 dist-test-slave-4tdj krb5kdc[24594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487048, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.67@KRBTEST.COM
I20260811 22:24:10.347800 27291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.333985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:35367 (local address 127.29.3.65:60995)
0811 22:24:10.334550 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:10.334565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:10.334668 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:10.336055 (+ 1387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:10.336061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:10.336619 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:10.337109 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:10.337124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.338421 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.338425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:10.339448 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:10.339459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.340486 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.340489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:10.340554 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:10.341197 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:10.341218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:10.343305 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:10.345343 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:10.345346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:10.345348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:10.345586 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:10.345923 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:10.345927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:10.345929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:10.346087 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:10.346770 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:10.346777 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:10.347038 (+ 261us) client_negotiation.cc:769] Sending connection context
0811 22:24:10.347213 (+ 175us) client_negotiation.cc:240] Negotiation successful
0811 22:24:10.347599 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":289,"spinlock_wait_cycles":45184,"thread_start_us":86,"threads_started":1}
I20260811 22:24:10.347810 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.334136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:47913 (local address 127.29.3.66:45681)
0811 22:24:10.334525 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:10.334552 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:10.334594 (+ 42us) server_negotiation.cc:407] Connection header received
0811 22:24:10.335304 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:10.335309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:10.335672 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:10.335946 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:10.337291 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.338252 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.339564 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.340139 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.340299 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:10.343251 (+ 2952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:10.343259 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:10.343278 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:10.343313 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:10.345079 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:10.345716 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:10.345722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:10.345726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:10.345858 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:10.346191 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:10.346194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:10.346195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:10.346661 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:10.346783 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:10.347348 (+ 565us) server_negotiation.cc:299] Negotiation successful
0811 22:24:10.347610 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":137,"threads_started":1}
I20260811 22:24:10.347810 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.334237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:60995 (local address 127.29.3.67:35367)
0811 22:24:10.334645 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:10.334668 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:10.334732 (+ 64us) server_negotiation.cc:407] Connection header received
0811 22:24:10.335383 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:10.335403 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:10.335717 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:10.335979 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:10.337280 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.338274 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.339584 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.340193 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.340344 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:10.343486 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:10.343491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:10.343503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:10.343540 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:10.345205 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:10.345696 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:10.345702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:10.345708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:10.345830 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:10.346178 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:10.346182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:10.346183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:10.346653 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:10.346775 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:10.347345 (+ 570us) server_negotiation.cc:299] Negotiation successful
0811 22:24:10.347613 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":149,"threads_started":1}
I20260811 22:24:10.347823 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.333945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45681 (local address 127.29.3.65:47913)
0811 22:24:10.334079 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:10.334093 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:10.334205 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:10.336056 (+ 1851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:10.336061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:10.336617 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:10.337109 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:10.337124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.338392 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.338396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:10.339449 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:10.339460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:10.340268 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:10.340272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:10.340444 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:10.341250 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:10.341263 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:10.342985 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:10.345214 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:10.345234 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:10.345248 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:10.345583 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:10.345959 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:10.345961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:10.345963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:10.346098 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:10.346772 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:10.346779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:10.347034 (+ 255us) client_negotiation.cc:769] Sending connection context
0811 22:24:10.347212 (+ 178us) client_negotiation.cc:240] Negotiation successful
0811 22:24:10.347675 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":296,"spinlock_wait_cycles":13184}
W20260811 22:24:10.350896 26637 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
W20260811 22:24:10.351063 26636 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:24:10.351177 26636 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92; no voters: 532715aa46b244a2a96f3cc396e7eef3, a5d58361fb004a7cb54d23491d52c821
I20260811 22:24:10.351498 27290 raft_consensus.cc:2726] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:24:10.920954 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:10.920504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:50221)
0811 22:24:10.920790 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:10.920852 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":86,"threads_started":1}
I20260811 22:24:11.081349 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:11.080879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:33157)
0811 22:24:11.081179 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:11.081248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260811 22:24:11.291126 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:11.290728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:59847)
0811 22:24:11.290967 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:11.291027 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260811 22:24:11.922287 27300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:11.921827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:44955)
0811 22:24:11.922113 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:11.922181 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260811 22:24:12.005038 27301 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:12.005182 27301 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:12.005646 27301 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
W20260811 22:24:12.006467 26637 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
W20260811 22:24:12.006585 26636 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:24:12.006676 26636 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92; no voters: 532715aa46b244a2a96f3cc396e7eef3, a5d58361fb004a7cb54d23491d52c821
I20260811 22:24:12.006835 27301 raft_consensus.cc:2726] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:24:12.082623 27305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:12.082126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:37999)
0811 22:24:12.082454 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:12.082520 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":94,"threads_started":1}
I20260811 22:24:12.292457 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:12.291925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:40127)
0811 22:24:12.292265 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:12.292337 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":97,"threads_started":1}
I20260811 22:24:12.923692 27307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:12.923156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:42975)
0811 22:24:12.923518 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:12.923588 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":103,"threads_started":1}
I20260811 22:24:13.083930 27308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:13.083453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:33301)
0811 22:24:13.083766 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:13.083830 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260811 22:24:13.293833 27310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:13.293346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:33025)
0811 22:24:13.293662 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:13.293718 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":78,"threads_started":1}
I20260811 22:24:13.925267 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:13.924733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:58583)
0811 22:24:13.925076 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:13.925136 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":93,"threads_started":1}
I20260811 22:24:14.085268 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:14.084780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:58825)
0811 22:24:14.085076 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:14.085145 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260811 22:24:14.295248 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:14.294758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:48155)
0811 22:24:14.295069 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:14.295141 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260811 22:24:14.536348 27315 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:14.536522 27315 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:14.536965 27315 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
W20260811 22:24:14.537725 26636 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
W20260811 22:24:14.537847 26637 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:24:14.537911 26637 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92; no voters: 532715aa46b244a2a96f3cc396e7eef3, a5d58361fb004a7cb54d23491d52c821
I20260811 22:24:14.538080 27315 raft_consensus.cc:2726] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:24:14.926662 27316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:14.926164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:42505)
0811 22:24:14.926490 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:14.926560 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":80,"threads_started":1}
I20260811 22:24:15.086585 27317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:15.086059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:35617)
0811 22:24:15.086412 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:15.086479 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":82,"threads_started":1}
I20260811 22:24:15.296614 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:15.296128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:47571)
0811 22:24:15.296446 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:15.296510 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":97,"threads_started":1}
I20260811 22:24:15.927948 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:15.927462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:39135)
0811 22:24:15.927778 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:15.927847 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":91,"threads_started":1}
I20260811 22:24:16.087942 27323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:16.087429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:45271)
0811 22:24:16.087760 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:16.087828 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":85,"threads_started":1}
I20260811 22:24:16.298004 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:16.297497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:34727)
0811 22:24:16.297831 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:16.297901 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":82,"threads_started":1}
I20260811 22:24:16.929173 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:16.928758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:35299)
0811 22:24:16.929011 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:16.929071 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":81,"threads_started":1}
I20260811 22:24:17.089309 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:17.088797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:33613)
0811 22:24:17.089130 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:17.089196 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":113,"threads_started":1}
I20260811 22:24:17.299350 27329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:17.298891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:40805)
0811 22:24:17.299177 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:17.299245 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":79,"threads_started":1}
I20260811 22:24:17.930438 27330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:17.929953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:38193)
0811 22:24:17.930225 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:17.930325 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260811 22:24:18.090713 27331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:18.090198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:37239)
0811 22:24:18.090538 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:18.090607 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":141,"threads_started":1}
I20260811 22:24:18.300570 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:18.300135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:33049)
0811 22:24:18.300402 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:18.300468 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260811 22:24:18.931941 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:18.931486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:51627)
0811 22:24:18.931777 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:18.931838 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":99,"threads_started":1}
I20260811 22:24:19.092165 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:19.091660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:41595)
0811 22:24:19.091983 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:19.092060 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
I20260811 22:24:19.301878 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:19.301397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:54961)
0811 22:24:19.301669 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:19.301774 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260811 22:24:19.410917 27338 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:19.411063 27338 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:19.411463 27338 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
W20260811 22:24:19.412313 26636 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
W20260811 22:24:19.412431 26637 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:24:19.412511 26637 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92; no voters: 532715aa46b244a2a96f3cc396e7eef3, a5d58361fb004a7cb54d23491d52c821
I20260811 22:24:19.412654 27338 raft_consensus.cc:2726] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:24:19.933131 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:19.932674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:46679)
0811 22:24:19.932945 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:19.933026 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":85,"threads_started":1}
I20260811 22:24:20.093492 27341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:20.093045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:52243)
0811 22:24:20.093320 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:20.093385 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260811 22:24:20.303241 27344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:20.302754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:32929)
0811 22:24:20.303052 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:20.303122 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260811 22:24:20.934440 27345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:20.933952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:42563)
0811 22:24:20.934241 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:20.934336 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260811 22:24:21.094810 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:21.094282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:59461)
0811 22:24:21.094630 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:21.094700 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":97,"threads_started":1}
I20260811 22:24:21.304617 27349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:21.304088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:38939)
0811 22:24:21.304437 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:21.304508 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":111,"threads_started":1}
I20260811 22:24:21.935736 27354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:21.935230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:56773)
0811 22:24:21.935547 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:21.935616 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":86,"threads_started":1}
I20260811 22:24:22.096041 27355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:22.095541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:46451)
0811 22:24:22.095859 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:22.095928 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":81,"threads_started":1}
I20260811 22:24:22.306109 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:22.305625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:38671)
0811 22:24:22.305925 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:22.306003 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":103,"threads_started":1}
I20260811 22:24:22.937028 27357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:22.936541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:40313)
0811 22:24:22.936855 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:22.936923 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":82,"threads_started":1}
I20260811 22:24:23.097285 27361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:23.096862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:34301)
0811 22:24:23.097122 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:23.097182 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
I20260811 22:24:23.241794 27362 raft_consensus.cc:482] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:23.241936 27362 raft_consensus.cc:504] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "532715aa46b244a2a96f3cc396e7eef3" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45681 } } peers { permanent_uuid: "a5d58361fb004a7cb54d23491d52c821" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 35367 } } peers { permanent_uuid: "8f3055bb4c254788a73c91ddc8ee5e92" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 35483 } }
I20260811 22:24:23.242451 27362 leader_election.cc:288] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681), a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367)
W20260811 22:24:23.243173 26637 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a5d58361fb004a7cb54d23491d52c821 (127.29.3.67:35367): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
W20260811 22:24:23.243299 26636 leader_election.cc:341] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 532715aa46b244a2a96f3cc396e7eef3 (127.29.3.66:45681): Not found: Tablet not found: b8a92cd691e34c9ebad7a0fe15932a9b
I20260811 22:24:23.243374 26636 leader_election.cc:302] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f3055bb4c254788a73c91ddc8ee5e92; no voters: 532715aa46b244a2a96f3cc396e7eef3, a5d58361fb004a7cb54d23491d52c821
I20260811 22:24:23.243525 27362 raft_consensus.cc:2726] T b8a92cd691e34c9ebad7a0fe15932a9b P 8f3055bb4c254788a73c91ddc8ee5e92 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 22:24:23.307556 27364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:23.307004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:49123)
0811 22:24:23.307349 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:23.307422 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":90,"threads_started":1}
I20260811 22:24:23.938606 27366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:23.938072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.65:36789)
0811 22:24:23.938416 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:23.938476 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":87,"threads_started":1}
I20260811 22:24:24.098783 27367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:24.098283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.66:47761)
0811 22:24:24.098587 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:24.098651 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260811 22:24:24.300956 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 26543
I20260811 22:24:24.311120 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:24.308727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:38759 (local address 127.29.3.67:37179)
0811 22:24:24.310939 (+ 2212us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:24.311014 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.126:38759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2108,"thread_start_us":1708,"threads_started":1}
I20260811 22:24:24.318215 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 26838
I20260811 22:24:24.337489 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 27069
2026-08-11T22:24:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59616 ms)
[----------] 30 tests from SecurityITest (253411 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:24 dist-test-slave-4tdj krb5kdc[27374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:24 dist-test-slave-4tdj krb5kdc[27374](info): set up 2 sockets
Aug 11 22:24:24 dist-test-slave-4tdj krb5kdc[27374](info): commencing operation
krb5kdc: starting...
W20260811 22:24:25.565161 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:25 dist-test-slave-4tdj krb5kdc[27374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:24:25.697716 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:41541
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:41541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:25.814658 27393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:25.814973 27393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:25.815033 27393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:25.819098 27393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:25.819173 27393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:25.819201 27393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:25.819222 27393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:25.819244 27393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:25.824852 27393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:41541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:41541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:25.826094 27393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:25.827246 27393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:25.836771 27398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:25.836778 27399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:25.836838 27393 server_base.cc:1034] running on GCE node
W20260811 22:24:25.836794 27406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:25.837663 27393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:25.838889 27393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:25.840046 27393 hybrid_clock.cc:648] HybridClock initialized: now 1786487065840019 us; error 43 us; skew 500 ppm
Aug 11 22:24:25 dist-test-slave-4tdj krb5kdc[27374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487065, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:25.842985 27393 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:25.843994 27393 webserver.cc:460] Webserver started at http://127.29.3.126:36745/ using document root <none> and password file <none>
I20260811 22:24:25.844576 27393 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:25.844666 27393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:25.844889 27393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:25.847091 27393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "f19448eca45b47a386c6ef45688f520d"
format_stamp: "Formatted at 2026-08-11 22:24:25 on dist-test-slave-4tdj"
server_key: "67ed9fc58d8f05892a44e60cc7672892"
server_key_iv: "91fb0dec6c369c0ec352021c04b70046"
server_key_version: "encryptionkey@0"
I20260811 22:24:25.847661 27393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "f19448eca45b47a386c6ef45688f520d"
format_stamp: "Formatted at 2026-08-11 22:24:25 on dist-test-slave-4tdj"
server_key: "67ed9fc58d8f05892a44e60cc7672892"
server_key_iv: "91fb0dec6c369c0ec352021c04b70046"
server_key_version: "encryptionkey@0"
I20260811 22:24:25.851646 27393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:24:25.854445 27427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:25.855831 27393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:25.855967 27393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "f19448eca45b47a386c6ef45688f520d"
format_stamp: "Formatted at 2026-08-11 22:24:25 on dist-test-slave-4tdj"
server_key: "67ed9fc58d8f05892a44e60cc7672892"
server_key_iv: "91fb0dec6c369c0ec352021c04b70046"
server_key_version: "encryptionkey@0"
I20260811 22:24:25.856087 27393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:25.878691 27393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:25.882169 27393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:25.882406 27393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:25.890071 27393 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:41541
I20260811 22:24:25.890156 27515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:41541 every 8 connection(s)
I20260811 22:24:25.891182 27393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:25.894198 27516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:25.895747 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 27393
I20260811 22:24:25.895866 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:25.896139 29709 external_mini_cluster.cc:1428] Setting key 4dc7b5efa7a52fa3006ecc26ed4d02b8
I20260811 22:24:25.901242 27516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d: Bootstrap starting.
I20260811 22:24:25.904125 27516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:25.904969 27516 log.cc:824] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:24:25 dist-test-slave-4tdj krb5kdc[27374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:25.907437 27516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d: No bootstrap required, opened a new log
I20260811 22:24:25.911507 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:25.897607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.29.3.126:41541)
0811 22:24:25.898067 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:25.898088 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:25.898135 (+ 47us) server_negotiation.cc:407] Connection header received
0811 22:24:25.899036 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:25.899063 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:25.899512 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:25.899917 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:25.900577 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:25.901996 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:25.902980 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:25.903673 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:25.903846 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:25.906465 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:25.906471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:25.906486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:25.906514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:25.908299 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:25.908844 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:25.908849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:25.908855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:25.908984 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:25.909261 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:25.909264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:25.909265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:25.909740 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:25.909898 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:25.910284 (+ 386us) server_negotiation.cc:299] Negotiation successful
0811 22:24:25.910675 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":127,"threads_started":1}
I20260811 22:24:25.911795 27516 raft_consensus.cc:348] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19448eca45b47a386c6ef45688f520d" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41541 } }
I20260811 22:24:25.912086 27516 raft_consensus.cc:374] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:25.912149 27516 raft_consensus.cc:729] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19448eca45b47a386c6ef45688f520d, State: Initialized, Role: FOLLOWER
I20260811 22:24:25.912662 27516 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19448eca45b47a386c6ef45688f520d" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41541 } }
I20260811 22:24:25.912835 27516 raft_consensus.cc:388] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:25.912904 27516 raft_consensus.cc:482] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:25.913007 27516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:25.914268 27516 raft_consensus.cc:504] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19448eca45b47a386c6ef45688f520d" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41541 } }
I20260811 22:24:25.914662 27516 leader_election.cc:302] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f19448eca45b47a386c6ef45688f520d; no voters:
I20260811 22:24:25.914990 27516 leader_election.cc:288] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:25.915153 27525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:25.915366 27525 raft_consensus.cc:686] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [term 1 LEADER]: Becoming Leader. State: Replica: f19448eca45b47a386c6ef45688f520d, State: Running, Role: LEADER
I20260811 22:24:25.915971 27525 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19448eca45b47a386c6ef45688f520d" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41541 } }
I20260811 22:24:25.916291 27516 sys_catalog.cc:564] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:25.917753 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f19448eca45b47a386c6ef45688f520d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19448eca45b47a386c6ef45688f520d" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41541 } } }
I20260811 22:24:25.917796 27532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19448eca45b47a386c6ef45688f520d. Latest consensus state: current_term: 1 leader_uuid: "f19448eca45b47a386c6ef45688f520d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19448eca45b47a386c6ef45688f520d" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41541 } } }
I20260811 22:24:25.917891 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:25.917893 27532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:25.918412 27546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:25.921787 27546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:25.927721 27546 catalog_manager.cc:1347] Generated new cluster ID: 9bb53830a86d4f7a9e4e36a603081d7c
I20260811 22:24:25.927796 27546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:25.940068 27546 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:25.941169 27546 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:25.951754 27546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d: Generated new TSK 0
I20260811 22:24:25.952783 27546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:24:25.994426 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41541
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:26.115725 27559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:26.116037 27559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:26.116098 27559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:26.120486 27559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:26.120579 27559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:26.120643 27559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:26.126356 27559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:26.127566 27559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:26.128650 27559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:26.139057 27571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:26.139041 27568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:26.139221 27559 server_base.cc:1034] running on GCE node
W20260811 22:24:26.139133 27567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:26.139786 27559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:26.140481 27559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:26.141691 27559 hybrid_clock.cc:648] HybridClock initialized: now 1786487066141666 us; error 35 us; skew 500 ppm
Aug 11 22:24:26 dist-test-slave-4tdj krb5kdc[27374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487066, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:26.144824 27559 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:24:26.145833 27559 webserver.cc:460] Webserver started at http://127.29.3.65:45121/ using document root <none> and password file <none>
I20260811 22:24:26.146474 27559 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:26.146545 27559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:26.146762 27559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:26.149055 27559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "96e8ef5a3bd8d1ded9bda6221f3d58d2"
server_key_iv: "6491dc02b69ba8dff68da9ef2af3bb0d"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.149714 27559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "96e8ef5a3bd8d1ded9bda6221f3d58d2"
server_key_iv: "6491dc02b69ba8dff68da9ef2af3bb0d"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.154012 27559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:26.156816 27579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.158362 27559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 22:24:26.158512 27559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "96e8ef5a3bd8d1ded9bda6221f3d58d2"
server_key_iv: "6491dc02b69ba8dff68da9ef2af3bb0d"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.158641 27559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:26.187678 27559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:26.191504 27559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:26.191736 27559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:26.192466 27559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:26.193403 27559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:26.193454 27559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.193512 27559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:26.193537 27559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.202853 27559 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:44649
I20260811 22:24:26.202884 27768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:44649 every 8 connection(s)
I20260811 22:24:26.203850 27559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:26.212718 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 27559
I20260811 22:24:26.212836 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:24:26.213116 29709 external_mini_cluster.cc:1428] Setting key bcc2c57011f2fbf4f3978c08351772f8
Aug 11 22:24:26 dist-test-slave-4tdj krb5kdc[27374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487066, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:26.218788 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.205915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:58777 (local address 127.29.3.126:41541)
0811 22:24:26.206070 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.206075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.207077 (+ 1002us) server_negotiation.cc:407] Connection header received
0811 22:24:26.208443 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.208446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.208498 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.208583 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:26.209839 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.210421 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.211593 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.211846 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.211932 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.214581 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:26.214585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:26.214588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:26.214617 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:26.216160 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.216807 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.216812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.216814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.216974 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.217439 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.217443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.217445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.217672 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:26.217804 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.218430 (+ 626us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.218593 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:24:26.219689 27772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.206399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:41541 (local address 127.29.3.65:58777)
0811 22:24:26.206911 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.206958 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.208176 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.208747 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.208754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.209133 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:26.209676 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.209691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.210522 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.210525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.211462 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.211472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.211973 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.211976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.212119 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.212740 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:26.212757 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:26.214407 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:26.216301 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:26.216308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:26.216322 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:26.216651 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:26.217138 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:26.217141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:26.217143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:26.217332 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:26.217790 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:26.217798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:26.218105 (+ 307us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.218385 (+ 280us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.218763 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":122,"threads_started":1}
I20260811 22:24:26.221203 27770 heartbeater.cc:347] Connected to a master server at 127.29.3.126:41541
I20260811 22:24:26.221470 27770 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:26.222038 27770 heartbeater.cc:511] Master 127.29.3.126:41541 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:24:26.223680 27453 ts_manager.cc:194] Registered new tserver with Master: 7d0d99aa7dfb4652b9ba19a36129c5ef (127.29.3.65:44649)
I20260811 22:24:26.224960 27453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:58777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:24:26.243367 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41541
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:26.361992 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:26.362371 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:26.362424 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:26.367070 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:26.367153 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:26.367216 27778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:26.373099 27778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:26.374250 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:26.375327 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:26.385754 27791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:26.385748 27794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:26.385748 27789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:26.385895 27778 server_base.cc:1034] running on GCE node
I20260811 22:24:26.386507 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:26.387210 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:26.388383 27778 hybrid_clock.cc:648] HybridClock initialized: now 1786487066388351 us; error 48 us; skew 500 ppm
Aug 11 22:24:26 dist-test-slave-4tdj krb5kdc[27374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487066, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:26.391438 27778 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:24:26.392467 27778 webserver.cc:460] Webserver started at http://127.29.3.66:42629/ using document root <none> and password file <none>
I20260811 22:24:26.393069 27778 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:26.393142 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:26.393366 27778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:26.395592 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "bde76b053f714e76aab5b74e0a2622a6"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "eba8eb74e8f711e95f44ea9ea4134daa"
server_key_iv: "ae7dc1a4e6ff06b1a605c34f04df3d14"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.396179 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "bde76b053f714e76aab5b74e0a2622a6"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "eba8eb74e8f711e95f44ea9ea4134daa"
server_key_iv: "ae7dc1a4e6ff06b1a605c34f04df3d14"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.400192 27778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:26.402760 27803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.404027 27778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:24:26.404160 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "bde76b053f714e76aab5b74e0a2622a6"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "eba8eb74e8f711e95f44ea9ea4134daa"
server_key_iv: "ae7dc1a4e6ff06b1a605c34f04df3d14"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.404269 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:26.430333 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:26.433879 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:26.434074 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:26.434790 27778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:26.435702 27778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:26.435753 27778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.435809 27778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:26.435837 27778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.444892 27778 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:36135
I20260811 22:24:26.444923 28013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:36135 every 8 connection(s)
I20260811 22:24:26.445858 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:26.451581 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 27778
I20260811 22:24:26.451720 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:24:26.452033 29709 external_mini_cluster.cc:1428] Setting key c182c15ec2dd3bc3756ec0b48e396780
Aug 11 22:24:26 dist-test-slave-4tdj krb5kdc[27374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487066, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:26.461064 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.447829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:51095 (local address 127.29.3.126:41541)
0811 22:24:26.447976 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.447979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.448891 (+ 912us) server_negotiation.cc:407] Connection header received
0811 22:24:26.450237 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.450240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.450330 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.450413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:26.451751 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.452357 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.453749 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.453977 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.454067 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.456824 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:26.456828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:26.456830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:26.456859 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:26.458376 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.459081 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.459086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.459088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.459190 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.459581 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.459584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.459586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.459861 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:26.459993 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.460793 (+ 800us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.460921 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:24:26.462417 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.448246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:41541 (local address 127.29.3.66:51095)
0811 22:24:26.448733 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.448779 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.449999 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.450615 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.450622 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.451009 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:26.451590 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.451608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.452503 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.452506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.453621 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.453632 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.454128 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.454133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.454347 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.454939 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:26.454957 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:26.456602 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:26.458500 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:26.458506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:26.458522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:26.458851 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:26.459285 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:26.459288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:26.459290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:26.459487 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:26.460064 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:26.460073 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:26.460469 (+ 396us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.460758 (+ 289us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.461166 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:24:26.463776 28017 heartbeater.cc:347] Connected to a master server at 127.29.3.126:41541
I20260811 22:24:26.464020 28017 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:26.464543 28017 heartbeater.cc:511] Master 127.29.3.126:41541 requested a full tablet report, sending...
I20260811 22:24:26.465713 27453 ts_manager.cc:194] Registered new tserver with Master: bde76b053f714e76aab5b74e0a2622a6 (127.29.3.66:36135)
I20260811 22:24:26.466527 27453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:51095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:24:26.485658 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41541
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:26.603767 28022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:26.604055 28022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:26.604099 28022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:26.608301 28022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:26.608387 28022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:26.608449 28022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:26.614187 28022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:26.615361 28022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:26.616396 28022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:26.626365 28037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:26.626323 28029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:26.626344 28022 server_base.cc:1034] running on GCE node
W20260811 22:24:26.626341 28031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:26.626993 28022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:26.627627 28022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:26.628791 28022 hybrid_clock.cc:648] HybridClock initialized: now 1786487066628760 us; error 44 us; skew 500 ppm
Aug 11 22:24:26 dist-test-slave-4tdj krb5kdc[27374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487066, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:26.631989 28022 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:24:26.633015 28022 webserver.cc:460] Webserver started at http://127.29.3.67:39685/ using document root <none> and password file <none>
I20260811 22:24:26.633620 28022 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:26.633716 28022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:26.633935 28022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:26.636094 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "7f6333655ea840aea2b9b2983ccf788f"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "5966b553c8bfeb45de42db5d7c8bb502"
server_key_iv: "22799872f59c96382154ca0401cb77f5"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.636620 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "7f6333655ea840aea2b9b2983ccf788f"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "5966b553c8bfeb45de42db5d7c8bb502"
server_key_iv: "22799872f59c96382154ca0401cb77f5"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.640527 28022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 22:24:26.643085 28047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.644340 28022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:24:26.644459 28022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "7f6333655ea840aea2b9b2983ccf788f"
format_stamp: "Formatted at 2026-08-11 22:24:26 on dist-test-slave-4tdj"
server_key: "5966b553c8bfeb45de42db5d7c8bb502"
server_key_iv: "22799872f59c96382154ca0401cb77f5"
server_key_version: "encryptionkey@0"
I20260811 22:24:26.644573 28022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:26.666325 28022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:26.669524 28022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:26.669735 28022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:26.670456 28022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:26.671409 28022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:26.671459 28022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.671506 28022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:26.671535 28022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:26.680742 28022 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:39843
I20260811 22:24:26.680773 28231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:39843 every 8 connection(s)
I20260811 22:24:26.681658 28022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:26.683892 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 28022
I20260811 22:24:26.684012 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:24:26.684300 29709 external_mini_cluster.cc:1428] Setting key 734c9f79e295c16ff468f17756a19f28
Aug 11 22:24:26 dist-test-slave-4tdj krb5kdc[27374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487066, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:26.695138 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.683537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:32799 (local address 127.29.3.126:41541)
0811 22:24:26.683736 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.683741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.684670 (+ 929us) server_negotiation.cc:407] Connection header received
0811 22:24:26.686106 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.686109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.686156 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.686235 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:26.687483 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.688018 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.689145 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.689365 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.689432 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.691836 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:26.691839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:26.691841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:26.691867 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:26.693131 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.693669 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.693671 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.693672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.693744 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.694022 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.694024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.694025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.694203 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:26.694273 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.694889 (+ 616us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.695008 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:24:26.695943 28237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.683964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:41541 (local address 127.29.3.67:32799)
0811 22:24:26.684494 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.684549 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.685885 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.686448 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.686458 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.686837 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:26.687350 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.687364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.688133 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.688136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.689040 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.689060 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.689470 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.689472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.689618 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.690139 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:26.690157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:26.691675 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:26.693275 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:26.693281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:26.693294 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:26.693560 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:26.693813 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:26.693815 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:26.693816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:26.693957 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:26.694327 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:26.694335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:26.694597 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.694798 (+ 201us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.695076 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":137,"threads_started":1}
I20260811 22:24:26.697196 28232 heartbeater.cc:347] Connected to a master server at 127.29.3.126:41541
I20260811 22:24:26.697482 28232 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:26.698115 28232 heartbeater.cc:511] Master 127.29.3.126:41541 requested a full tablet report, sending...
I20260811 22:24:26.699208 27453 ts_manager.cc:194] Registered new tserver with Master: 7f6333655ea840aea2b9b2983ccf788f (127.29.3.67:39843)
I20260811 22:24:26.699785 27453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:32799
I20260811 22:24:26.710860 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:24:26.720337 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.712966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.29.3.126:41541)
0811 22:24:26.713090 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.713094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.713259 (+ 165us) server_negotiation.cc:407] Connection header received
0811 22:24:26.713370 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.713372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.713417 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.713488 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:26.714173 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.714747 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.715597 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.715900 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.715982 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.717326 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:26.717329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:26.717331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:26.717357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:26.718852 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.719265 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.719269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.719270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.719360 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:26.719566 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:26.719568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:26.719569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:26.719805 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:26.719913 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.720081 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.720189 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 22:24:26.723407 27453 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:24:26.726543 27453 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:24:26.740901 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.735600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.29.3.65:44649)
0811 22:24:26.736042 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.736049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.736076 (+ 27us) server_negotiation.cc:407] Connection header received
0811 22:24:26.736270 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.736276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.736493 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.736658 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:26.737181 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.738203 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.739436 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.740200 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.740301 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.740464 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.740640 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.740743 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":129,"threads_started":1}
I20260811 22:24:26.740891 28248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.735427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44649 (local address 127.0.0.1:49022)
0811 22:24:26.735852 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.735912 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.736051 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.736848 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.736854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.736878 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:26.737025 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.737033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.738345 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.738348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.739285 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.739296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.740323 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.740327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.740444 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.740485 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.740593 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.740731 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":103,"threads_started":1}
I20260811 22:24:26.742225 28254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.735480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:36135 (local address 127.0.0.1:46022)
0811 22:24:26.736685 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.736698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.736798 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.737698 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.737701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.737719 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:26.737837 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.737843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.739299 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.739302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.740617 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.740629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.741825 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.741829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.741936 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.741961 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.742035 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.742108 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1143,"mutex_wait_us":26,"thread_start_us":103,"threads_started":1}
I20260811 22:24:26.742456 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.736228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:39843 (local address 127.0.0.1:38480)
0811 22:24:26.737192 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.737205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.737302 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.737953 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.737956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.737968 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:26.738036 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.738040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.739420 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.739423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.740617 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.740629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.742006 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.742010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.742097 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.742122 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.742201 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.742274 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":50,"thread_start_us":92,"threads_started":1}
I20260811 22:24:26.742525 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.736639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38480 (local address 127.29.3.67:39843)
0811 22:24:26.737038 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.737050 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.737304 (+ 254us) server_negotiation.cc:407] Connection header received
0811 22:24:26.737400 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.737404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.737558 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.737723 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:26.738141 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.739286 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.740766 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.741616 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.741735 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.741940 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.742253 (+ 313us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.742385 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":164,"threads_started":1}
I20260811 22:24:26.743032 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.736299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46022 (local address 127.29.3.66:36135)
0811 22:24:26.737247 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.737257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.737281 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:24:26.737362 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.737367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.737559 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.737719 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:26.737950 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.739130 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.740767 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.741703 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.741801 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.741915 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.742792 (+ 877us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.742893 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":86,"threads_started":1}
I20260811 22:24:26.744653 27657 tablet_service.cc:1467] Processing CreateTablet for tablet 82aacd135ac44b6ba04d279e9e67c6f8 (DEFAULT_TABLE table=test-table [id=1c9d299efb82471fb1283de337bb4107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:26.745710 27657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aacd135ac44b6ba04d279e9e67c6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:26.745414 28135 tablet_service.cc:1467] Processing CreateTablet for tablet 82aacd135ac44b6ba04d279e9e67c6f8 (DEFAULT_TABLE table=test-table [id=1c9d299efb82471fb1283de337bb4107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:26.745764 27890 tablet_service.cc:1467] Processing CreateTablet for tablet 82aacd135ac44b6ba04d279e9e67c6f8 (DEFAULT_TABLE table=test-table [id=1c9d299efb82471fb1283de337bb4107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:26.746593 28135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aacd135ac44b6ba04d279e9e67c6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:26.746804 27890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82aacd135ac44b6ba04d279e9e67c6f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:26.752780 28264 tablet_bootstrap.cc:492] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef: Bootstrap starting.
I20260811 22:24:26.754627 28268 tablet_bootstrap.cc:492] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Bootstrap starting.
I20260811 22:24:26.755541 28264 tablet_bootstrap.cc:654] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:26.755842 28267 tablet_bootstrap.cc:492] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Bootstrap starting.
I20260811 22:24:26.756521 28264 log.cc:824] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:26.757375 28268 tablet_bootstrap.cc:654] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:26.758275 28267 tablet_bootstrap.cc:654] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:26.758311 28268 log.cc:824] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:26.759297 28267 log.cc:824] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:26.759358 28264 tablet_bootstrap.cc:492] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef: No bootstrap required, opened a new log
I20260811 22:24:26.759565 28264 ts_tablet_manager.cc:1397] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:24:26.760200 28268 tablet_bootstrap.cc:492] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: No bootstrap required, opened a new log
I20260811 22:24:26.760412 28268 ts_tablet_manager.cc:1397] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:24:26.761641 28267 tablet_bootstrap.cc:492] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: No bootstrap required, opened a new log
I20260811 22:24:26.761919 28267 ts_tablet_manager.cc:1397] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260811 22:24:26.763826 28264 raft_consensus.cc:348] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.764076 28264 raft_consensus.cc:374] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:26.764133 28264 raft_consensus.cc:729] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0d99aa7dfb4652b9ba19a36129c5ef, State: Initialized, Role: FOLLOWER
I20260811 22:24:26.764611 28264 consensus_queue.cc:260] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.764554 28268 raft_consensus.cc:348] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.764838 28268 raft_consensus.cc:374] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:26.764897 28268 raft_consensus.cc:729] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde76b053f714e76aab5b74e0a2622a6, State: Initialized, Role: FOLLOWER
I20260811 22:24:26.765347 28268 consensus_queue.cc:260] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.765638 28264 ts_tablet_manager.cc:1428] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 22:24:26.766013 27770 heartbeater.cc:503] Master 127.29.3.126:41541 was elected leader, sending a full tablet report...
I20260811 22:24:26.766556 28268 ts_tablet_manager.cc:1428] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 22:24:26.766937 28017 heartbeater.cc:503] Master 127.29.3.126:41541 was elected leader, sending a full tablet report...
I20260811 22:24:26.767191 28267 raft_consensus.cc:348] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.767458 28267 raft_consensus.cc:374] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:26.767508 28267 raft_consensus.cc:729] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6333655ea840aea2b9b2983ccf788f, State: Initialized, Role: FOLLOWER
I20260811 22:24:26.767970 28267 consensus_queue.cc:260] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.768666 28232 heartbeater.cc:503] Master 127.29.3.126:41541 was elected leader, sending a full tablet report...
I20260811 22:24:26.769045 28267 ts_tablet_manager.cc:1428] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260811 22:24:26.807010 28273 raft_consensus.cc:482] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:26.807296 28273 raft_consensus.cc:504] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.808614 28273 leader_election.cc:288] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bde76b053f714e76aab5b74e0a2622a6 (127.29.3.66:36135), 7f6333655ea840aea2b9b2983ccf788f (127.29.3.67:39843)
I20260811 22:24:26.812706 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.809047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:57569 (local address 127.29.3.67:39843)
0811 22:24:26.809177 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.809182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.809376 (+ 194us) server_negotiation.cc:407] Connection header received
0811 22:24:26.809509 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.809513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.809565 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.809643 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:26.809935 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.810529 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.811752 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.812151 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.812234 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.812301 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.812505 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.812582 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:24:26.812724 28255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.808961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:49571 (local address 127.29.3.66:36135)
0811 22:24:26.809090 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.809095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.809487 (+ 392us) server_negotiation.cc:407] Connection header received
0811 22:24:26.809647 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.809650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.809729 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.809806 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:26.810075 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.810604 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.811715 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.812173 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.812273 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.812357 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.812542 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.812612 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:24:26.812722 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.808919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:39843 (local address 127.29.3.65:57569)
0811 22:24:26.809273 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.809286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.809409 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.809659 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.809662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.809677 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:26.809823 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.809831 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.810631 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.810635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.811630 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.811640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.812274 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.812277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.812374 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.812399 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.812470 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.812543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":17,"thread_start_us":136,"threads_started":1}
I20260811 22:24:26.813299 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.808919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:36135 (local address 127.29.3.65:49571)
0811 22:24:26.809369 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.809383 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:26.809508 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:26.809831 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:26.809833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:26.809848 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:26.809964 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.809971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.810704 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.810706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.811593 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:26.811603 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.812293 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.812297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:26.812401 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.812425 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:24:26.812495 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:24:26.813182 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":52,"thread_start_us":139,"threads_started":1}
I20260811 22:24:26.813462 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aacd135ac44b6ba04d279e9e67c6f8" candidate_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6333655ea840aea2b9b2983ccf788f" is_pre_election: true
I20260811 22:24:26.813818 28165 raft_consensus.cc:2454] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0d99aa7dfb4652b9ba19a36129c5ef in term 0.
I20260811 22:24:26.813891 27942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aacd135ac44b6ba04d279e9e67c6f8" candidate_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde76b053f714e76aab5b74e0a2622a6" is_pre_election: true
I20260811 22:24:26.814168 27942 raft_consensus.cc:2454] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d0d99aa7dfb4652b9ba19a36129c5ef in term 0.
I20260811 22:24:26.814378 27584 leader_election.cc:302] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0d99aa7dfb4652b9ba19a36129c5ef, 7f6333655ea840aea2b9b2983ccf788f; no voters:
I20260811 22:24:26.814697 28273 raft_consensus.cc:2781] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:26.814777 28273 raft_consensus.cc:482] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:26.814806 28273 raft_consensus.cc:3037] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:26.815864 28273 raft_consensus.cc:504] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.816223 28273 leader_election.cc:288] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [CANDIDATE]: Term 1 election: Requested vote from peers bde76b053f714e76aab5b74e0a2622a6 (127.29.3.66:36135), 7f6333655ea840aea2b9b2983ccf788f (127.29.3.67:39843)
I20260811 22:24:26.816565 28165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aacd135ac44b6ba04d279e9e67c6f8" candidate_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6333655ea840aea2b9b2983ccf788f"
I20260811 22:24:26.816606 27942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82aacd135ac44b6ba04d279e9e67c6f8" candidate_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bde76b053f714e76aab5b74e0a2622a6"
I20260811 22:24:26.816699 28165 raft_consensus.cc:3037] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:26.816702 27942 raft_consensus.cc:3037] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:26.817993 27942 raft_consensus.cc:2454] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0d99aa7dfb4652b9ba19a36129c5ef in term 1.
I20260811 22:24:26.817993 28165 raft_consensus.cc:2454] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d0d99aa7dfb4652b9ba19a36129c5ef in term 1.
I20260811 22:24:26.818343 27584 leader_election.cc:302] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d0d99aa7dfb4652b9ba19a36129c5ef, 7f6333655ea840aea2b9b2983ccf788f; no voters:
I20260811 22:24:26.818522 28273 raft_consensus.cc:2781] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:26.818760 28273 raft_consensus.cc:686] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [term 1 LEADER]: Becoming Leader. State: Replica: 7d0d99aa7dfb4652b9ba19a36129c5ef, State: Running, Role: LEADER
I20260811 22:24:26.819106 28273 consensus_queue.cc:237] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } }
I20260811 22:24:26.822217 27453 catalog_manager.cc:5495] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d0d99aa7dfb4652b9ba19a36129c5ef (127.29.3.65). New cstate: current_term: 1 leader_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0d99aa7dfb4652b9ba19a36129c5ef" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 44649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 } health_report { overall_health: UNKNOWN } } }
I20260811 22:24:26.841684 28252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.838441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49028 (local address 127.29.3.65:44649)
0811 22:24:26.838604 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:26.838608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:26.838643 (+ 35us) server_negotiation.cc:407] Connection header received
0811 22:24:26.838726 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:26.838729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:26.838788 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:26.838869 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:24:26.839078 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.839623 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.840394 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:26.840617 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:26.840687 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:26.840920 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:24:26.841340 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:24:26.841430 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:26.841458 (+ 28us) server_negotiation.cc:299] Negotiation successful
0811 22:24:26.841509 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:24:26.847919 28165 raft_consensus.cc:1261] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 1 FOLLOWER]: Refusing update from remote peer 7d0d99aa7dfb4652b9ba19a36129c5ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:26.847919 27942 raft_consensus.cc:1261] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 1 FOLLOWER]: Refusing update from remote peer 7d0d99aa7dfb4652b9ba19a36129c5ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:26.848745 28283 consensus_queue.cc:1035] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bde76b053f714e76aab5b74e0a2622a6" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 36135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:26.848946 28273 consensus_queue.cc:1035] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7d0d99aa7dfb4652b9ba19a36129c5ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6333655ea840aea2b9b2983ccf788f" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:26.855957 28286 mvcc.cc:205] Tried to move back new op lower bound from 7317451025801035776 to 7317451025693990912. Current Snapshot: MvccSnapshot[applied={T|T < 7317451025801035776}]
I20260811 22:24:26.856663 28289 mvcc.cc:205] Tried to move back new op lower bound from 7317451025801035776 to 7317451025693990912. Current Snapshot: MvccSnapshot[applied={T|T < 7317451025801035776}]
I20260811 22:24:26.866184 27453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:26.866438 27453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:26.869246 27453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f19448eca45b47a386c6ef45688f520d: Sending DeleteTablet for 3 replicas of tablet 82aacd135ac44b6ba04d279e9e67c6f8
I20260811 22:24:26.870249 27657 tablet_service.cc:1514] Processing DeleteTablet for tablet 82aacd135ac44b6ba04d279e9e67c6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:26 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:49022
I20260811 22:24:26.870333 27890 tablet_service.cc:1514] Processing DeleteTablet for tablet 82aacd135ac44b6ba04d279e9e67c6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:26 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:46022
I20260811 22:24:26.870338 28135 tablet_service.cc:1514] Processing DeleteTablet for tablet 82aacd135ac44b6ba04d279e9e67c6f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:26 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:38480
I20260811 22:24:26.871040 28307 tablet_replica.cc:330] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: stopping tablet replica
I20260811 22:24:26.871435 28307 raft_consensus.cc:2229] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:26.871570 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 27559
I20260811 22:24:26.871821 28307 raft_consensus.cc:2258] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:26.874724 28306 tablet_replica.cc:330] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: stopping tablet replica
I20260811 22:24:26.875010 28306 raft_consensus.cc:2229] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:26.875280 28306 raft_consensus.cc:2258] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:26.884626 28306 ts_tablet_manager.cc:1905] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:26.886082 28307 ts_tablet_manager.cc:1905] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:26.887904 28306 ts_tablet_manager.cc:1918] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:26.888025 28306 log.cc:1196] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/82aacd135ac44b6ba04d279e9e67c6f8
I20260811 22:24:26.888377 28306 ts_tablet_manager.cc:1939] T 82aacd135ac44b6ba04d279e9e67c6f8 P 7f6333655ea840aea2b9b2983ccf788f: Deleting consensus metadata
I20260811 22:24:26.888852 28307 ts_tablet_manager.cc:1918] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:26.888954 28307 log.cc:1196] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/82aacd135ac44b6ba04d279e9e67c6f8
I20260811 22:24:26.889235 28307 ts_tablet_manager.cc:1939] T 82aacd135ac44b6ba04d279e9e67c6f8 P bde76b053f714e76aab5b74e0a2622a6: Deleting consensus metadata
I20260811 22:24:26.889849 27430 catalog_manager.cc:4841] TS 7f6333655ea840aea2b9b2983ccf788f (127.29.3.67:39843): tablet 82aacd135ac44b6ba04d279e9e67c6f8 (table test-table [id=1c9d299efb82471fb1283de337bb4107]) successfully deleted
I20260811 22:24:26.890538 27428 catalog_manager.cc:4841] TS bde76b053f714e76aab5b74e0a2622a6 (127.29.3.66:36135): tablet 82aacd135ac44b6ba04d279e9e67c6f8 (table test-table [id=1c9d299efb82471fb1283de337bb4107]) successfully deleted
W20260811 22:24:26.894861 27430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:44649 (error 108)
I20260811 22:24:26.895671 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 27778
I20260811 22:24:26.895928 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:26.895659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:44649 (local address 127.0.0.1:49032)
0811 22:24:26.895790 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:26.895853 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:44649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 22:24:26.907939 27430 catalog_manager.cc:4584] TS 7d0d99aa7dfb4652b9ba19a36129c5ef (127.29.3.65:44649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82aacd135ac44b6ba04d279e9e67c6f8: Network error: Client connection negotiation failed: client connection to 127.29.3.65:44649: connect: Connection refused (error 111)
I20260811 22:24:26.919365 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 28022
I20260811 22:24:26.946945 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 27393
2026-08-11T22:24:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2604 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T22:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:26Z Disabled control of system clock
I20260811 22:24:26.989172 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44661
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:44661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:24:27.109045 28315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:27.109364 28315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:27.109402 28315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:27.113498 28315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:27.113579 28315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:27.113592 28315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:27.113607 28315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:27.113629 28315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:27.118712 28315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:44661
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:44661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:27.119889 28315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:27.121287 28315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 22:24:27.130075 28315 server_base.cc:1034] running on GCE node
W20260811 22:24:27.130091 28322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.130046 28323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.130326 28325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:27.130990 28315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:27.132182 28315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:27.133351 28315 hybrid_clock.cc:648] HybridClock initialized: now 1786487067133324 us; error 41 us; skew 500 ppm
I20260811 22:24:27.138229 28315 webserver.cc:460] Webserver started at http://127.29.3.126:37351/ using document root <none> and password file <none>
I20260811 22:24:27.138921 28315 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:27.139000 28315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:27.139213 28315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:27.141415 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "ec19341f3efb421f9788b64e459d546e"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "e2832b9beb55858d0302c09161b77d62"
server_key_iv: "ca2baeca23e5b45bc9105b575fbc9307"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.142009 28315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "ec19341f3efb421f9788b64e459d546e"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "e2832b9beb55858d0302c09161b77d62"
server_key_iv: "ca2baeca23e5b45bc9105b575fbc9307"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.146273 28315 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:27.149201 28336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.150964 28315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260811 22:24:27.151149 28315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "ec19341f3efb421f9788b64e459d546e"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "e2832b9beb55858d0302c09161b77d62"
server_key_iv: "ca2baeca23e5b45bc9105b575fbc9307"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.151279 28315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:27.165335 28315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:27.166126 28315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:27.166360 28315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:27.173586 28315 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:44661
I20260811 22:24:27.173619 28420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:44661 every 8 connection(s)
I20260811 22:24:27.174677 28315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:27.176301 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 28315
I20260811 22:24:27.176416 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:27.176740 29709 external_mini_cluster.cc:1428] Setting key c8a901b1c17fafa72928eabb4b9d5748
I20260811 22:24:27.178691 28421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:27.184686 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e: Bootstrap starting.
I20260811 22:24:27.186475 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.178367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.29.3.126:44661)
0811 22:24:27.179153 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.179171 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.179212 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:24:27.179892 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.179910 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.180337 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.180669 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:27.181117 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.182095 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.183052 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.183541 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.183703 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.184582 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:27.184587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:27.184600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:27.185029 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:27.185102 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:27.185124 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:27.185211 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.185369 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.185658 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":168,"threads_started":1}
I20260811 22:24:27.187252 28421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:27.188216 28421 log.cc:824] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:27.190528 28421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e: No bootstrap required, opened a new log
I20260811 22:24:27.194603 28421 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19341f3efb421f9788b64e459d546e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44661 } }
I20260811 22:24:27.194885 28421 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:27.194936 28421 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec19341f3efb421f9788b64e459d546e, State: Initialized, Role: FOLLOWER
I20260811 22:24:27.195374 28421 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19341f3efb421f9788b64e459d546e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44661 } }
I20260811 22:24:27.195518 28421 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:27.195591 28421 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:27.195677 28421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:27.196673 28421 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19341f3efb421f9788b64e459d546e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44661 } }
I20260811 22:24:27.197065 28421 leader_election.cc:302] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec19341f3efb421f9788b64e459d546e; no voters:
I20260811 22:24:27.197468 28421 leader_election.cc:288] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:27.197566 28429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:27.197796 28429 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [term 1 LEADER]: Becoming Leader. State: Replica: ec19341f3efb421f9788b64e459d546e, State: Running, Role: LEADER
I20260811 22:24:27.198127 28429 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19341f3efb421f9788b64e459d546e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44661 } }
I20260811 22:24:27.198836 28421 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:27.199997 28430 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec19341f3efb421f9788b64e459d546e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19341f3efb421f9788b64e459d546e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44661 } } }
I20260811 22:24:27.200031 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec19341f3efb421f9788b64e459d546e. Latest consensus state: current_term: 1 leader_uuid: "ec19341f3efb421f9788b64e459d546e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19341f3efb421f9788b64e459d546e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 44661 } } }
I20260811 22:24:27.200153 28430 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:27.200155 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:27.200570 28447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:27.203485 28447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:27.210120 28447 catalog_manager.cc:1347] Generated new cluster ID: 8422dd6ee74a4d92a88e7a584deb4a53
I20260811 22:24:27.210201 28447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:27.228281 28447 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:27.229329 28447 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:27.236691 28447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e: Generated new TSK 0
I20260811 22:24:27.237486 28447 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:27.242380 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44661
--builtin_ntp_servers=127.29.3.84:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:24:27.378656 28466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:27.378949 28466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:27.378990 28466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:27.383318 28466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:27.383391 28466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:27.383469 28466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:27.388496 28466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:27.389732 28466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:27.391332 28466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:27.401495 28475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.401484 28472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.401480 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:27.402070 28466 server_base.cc:1034] running on GCE node
I20260811 22:24:27.402508 28466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:27.403199 28466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:27.404372 28466 hybrid_clock.cc:648] HybridClock initialized: now 1786487067404351 us; error 34 us; skew 500 ppm
I20260811 22:24:27.406451 28466 webserver.cc:460] Webserver started at http://127.29.3.65:35277/ using document root <none> and password file <none>
I20260811 22:24:27.407063 28466 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:27.407135 28466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:27.407375 28466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:27.409548 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "3c396acc3455493c8c12bc6e86a37617"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "16e09eb0a7991c7366ed808ecb3baa0f"
server_key_iv: "eb9d480960eea79f89d90caac17ccbaa"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.410105 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "3c396acc3455493c8c12bc6e86a37617"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "16e09eb0a7991c7366ed808ecb3baa0f"
server_key_iv: "eb9d480960eea79f89d90caac17ccbaa"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.413988 28466 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260811 22:24:27.416755 28489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.418149 28466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 22:24:27.418279 28466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "3c396acc3455493c8c12bc6e86a37617"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "16e09eb0a7991c7366ed808ecb3baa0f"
server_key_iv: "eb9d480960eea79f89d90caac17ccbaa"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.418433 28466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:27.451999 28466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:27.452783 28466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:27.452987 28466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:27.453634 28466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:27.454725 28466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:27.454787 28466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.454852 28466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:27.454876 28466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.464193 28466 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:41841
I20260811 22:24:27.464224 28690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:41841 every 8 connection(s)
I20260811 22:24:27.465166 28466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:27.472486 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 28466
I20260811 22:24:27.472612 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:24:27.472875 29709 external_mini_cluster.cc:1428] Setting key 3ccab49a8db336594cc7aaa4e1118025
I20260811 22:24:27.475041 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44661
--builtin_ntp_servers=127.29.3.84:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:24:27.475354 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.467063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:51745 (local address 127.29.3.126:44661)
0811 22:24:27.467236 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.467241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.468396 (+ 1155us) server_negotiation.cc:407] Connection header received
0811 22:24:27.469880 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.469883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.469944 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.470027 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:27.471228 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.471791 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.473141 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.473428 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.473542 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.474853 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:27.474857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:27.474859 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:27.474871 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:27.474891 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:27.474899 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:27.474954 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.475129 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.475208 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:24:27.476300 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.467503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44661 (local address 127.29.3.65:51745)
0811 22:24:27.468231 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.468281 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.469650 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.470280 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.470288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.470439 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:27.471071 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.471087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.471937 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.471940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.473006 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.473019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.473610 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.473613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.473825 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.474566 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:27.474569 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:27.474617 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:27.474693 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:27.474968 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:27.474979 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:27.474985 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.475123 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.475450 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":134,"threads_started":1}
I20260811 22:24:27.477502 28692 heartbeater.cc:347] Connected to a master server at 127.29.3.126:44661
I20260811 22:24:27.477803 28692 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:27.478438 28692 heartbeater.cc:511] Master 127.29.3.126:44661 requested a full tablet report, sending...
I20260811 22:24:27.480109 28363 ts_manager.cc:194] Registered new tserver with Master: 3c396acc3455493c8c12bc6e86a37617 (127.29.3.65:41841)
I20260811 22:24:27.481273 28363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.65:51745
W20260811 22:24:27.593362 28696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:27.593667 28696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:27.593724 28696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:27.597834 28696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:27.597893 28696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:27.597971 28696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:27.603031 28696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:27.604183 28696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:27.605650 28696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:27.615464 28706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.615484 28708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:27.615573 28696 server_base.cc:1034] running on GCE node
W20260811 22:24:27.615469 28704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:27.616253 28696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:27.616876 28696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:27.618034 28696 hybrid_clock.cc:648] HybridClock initialized: now 1786487067618008 us; error 50 us; skew 500 ppm
I20260811 22:24:27.620045 28696 webserver.cc:460] Webserver started at http://127.29.3.66:33649/ using document root <none> and password file <none>
I20260811 22:24:27.620615 28696 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:27.620675 28696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:27.620850 28696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:27.623054 28696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "cbd7babd74924d02a91c40ec1b7ec522"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "696a523ea80b6bb5cc39f2a4dc199373"
server_key_iv: "67983b7b6ee61e2ff7f0c58c602c46b0"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.623597 28696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "cbd7babd74924d02a91c40ec1b7ec522"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "696a523ea80b6bb5cc39f2a4dc199373"
server_key_iv: "67983b7b6ee61e2ff7f0c58c602c46b0"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.627405 28696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:24:27.630048 28717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.631425 28696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:27.631537 28696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "cbd7babd74924d02a91c40ec1b7ec522"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "696a523ea80b6bb5cc39f2a4dc199373"
server_key_iv: "67983b7b6ee61e2ff7f0c58c602c46b0"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.631640 28696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:27.652336 28696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:27.653052 28696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:27.653220 28696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:27.653797 28696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:27.654838 28696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:27.654897 28696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.654944 28696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:27.654964 28696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.663774 28696 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:33323
I20260811 22:24:27.663828 28903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:33323 every 8 connection(s)
I20260811 22:24:27.664697 28696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:27.674049 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 28696
I20260811 22:24:27.674216 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:24:27.674343 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.666634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:39499 (local address 127.29.3.126:44661)
0811 22:24:27.666803 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.666807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.667870 (+ 1063us) server_negotiation.cc:407] Connection header received
0811 22:24:27.669342 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.669345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.669396 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.669472 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:27.670490 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.671018 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.672191 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.672410 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.672483 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.673705 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:27.673710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:27.673712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:27.673730 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:27.673753 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:27.673761 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:27.673820 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.674062 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.674163 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 22:24:27.674556 29709 external_mini_cluster.cc:1428] Setting key 434078148221419fe613d88ef633b959
I20260811 22:24:27.675417 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.667023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44661 (local address 127.29.3.66:39499)
0811 22:24:27.667732 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.667774 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.669096 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.669659 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.669668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.669772 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:27.670356 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.670373 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.671127 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.671130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.672088 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.672098 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.672548 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.672551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.672711 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.673391 (+ 680us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:27.673393 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:27.673441 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:27.673519 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:27.673844 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:27.673856 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:27.673862 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.674016 (+ 154us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.674404 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":138,"threads_started":1}
I20260811 22:24:27.676791 28906 heartbeater.cc:347] Connected to a master server at 127.29.3.126:44661
I20260811 22:24:27.676926 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44661
--builtin_ntp_servers=127.29.3.84:40199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:24:27.677075 28906 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:27.677726 28906 heartbeater.cc:511] Master 127.29.3.126:44661 requested a full tablet report, sending...
I20260811 22:24:27.678982 28363 ts_manager.cc:194] Registered new tserver with Master: cbd7babd74924d02a91c40ec1b7ec522 (127.29.3.66:33323)
I20260811 22:24:27.679795 28363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.66:39499
W20260811 22:24:27.795336 28910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:27.795634 28910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:27.795674 28910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:27.799849 28910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:27.799913 28910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:27.799973 28910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:27.805014 28910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:44661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:27.806213 28910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:27.807698 28910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:27.817631 28924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.817783 28922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:27.817785 28920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:27.818080 28910 server_base.cc:1034] running on GCE node
I20260811 22:24:27.818535 28910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:27.819196 28910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:27.820380 28910 hybrid_clock.cc:648] HybridClock initialized: now 1786487067820349 us; error 42 us; skew 500 ppm
I20260811 22:24:27.822397 28910 webserver.cc:460] Webserver started at http://127.29.3.67:45835/ using document root <none> and password file <none>
I20260811 22:24:27.822979 28910 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:27.823045 28910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:27.823254 28910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:27.825480 28910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "8f89743b6b91417d985b8d0fb3ec8b09"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "09459eb1b23af8409bee3bc4ac510312"
server_key_iv: "473478e115bfe8d056bba0bc11f2f7bc"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.826061 28910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "8f89743b6b91417d985b8d0fb3ec8b09"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "09459eb1b23af8409bee3bc4ac510312"
server_key_iv: "473478e115bfe8d056bba0bc11f2f7bc"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.830019 28910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260811 22:24:27.832816 28933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.834112 28910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:27.834250 28910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "8f89743b6b91417d985b8d0fb3ec8b09"
format_stamp: "Formatted at 2026-08-11 22:24:27 on dist-test-slave-4tdj"
server_key: "09459eb1b23af8409bee3bc4ac510312"
server_key_iv: "473478e115bfe8d056bba0bc11f2f7bc"
server_key_version: "encryptionkey@0"
I20260811 22:24:27.834416 28910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:27.865892 28910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:27.866698 28910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:27.866885 28910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:27.867511 28910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:27.868539 28910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:27.868599 28910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.868656 28910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:27.868681 28910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:27.877544 28910 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:39847
I20260811 22:24:27.877583 29122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:39847 every 8 connection(s)
I20260811 22:24:27.878553 28910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:27.885505 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 28910
I20260811 22:24:27.885610 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:24:27.885862 29709 external_mini_cluster.cc:1428] Setting key 236fb49b9810d26ab1c411ee867b2938
I20260811 22:24:27.888200 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.880519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:38481 (local address 127.29.3.126:44661)
0811 22:24:27.880674 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.880677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.881730 (+ 1053us) server_negotiation.cc:407] Connection header received
0811 22:24:27.883264 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.883267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.883328 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.883410 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:27.884358 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.884918 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.886206 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.886472 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.886581 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.887717 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:27.887720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:27.887722 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:27.887734 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:27.887757 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:27.887765 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:27.887843 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.888013 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.888079 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:24:27.889114 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.880934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:44661 (local address 127.29.3.67:38481)
0811 22:24:27.881598 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.881641 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.883021 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.883566 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.883573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.883661 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:27.884201 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.884215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.885071 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.885074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.886081 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.886091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.886628 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.886632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.886810 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.887420 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:27.887421 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:27.887476 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:27.887561 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:27.887855 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:27.887866 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:27.887872 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.888009 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.888277 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260811 22:24:27.890280 29124 heartbeater.cc:347] Connected to a master server at 127.29.3.126:44661
I20260811 22:24:27.890525 29124 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:27.891072 29124 heartbeater.cc:511] Master 127.29.3.126:44661 requested a full tablet report, sending...
I20260811 22:24:27.892109 28363 ts_manager.cc:194] Registered new tserver with Master: 8f89743b6b91417d985b8d0fb3ec8b09 (127.29.3.67:39847)
I20260811 22:24:27.892720 28363 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.67:38481
I20260811 22:24:27.900162 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:24:27.906008 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.902177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44754 (local address 127.29.3.126:44661)
0811 22:24:27.902418 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.902421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.902494 (+ 73us) server_negotiation.cc:407] Connection header received
0811 22:24:27.902595 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.902597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.902641 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.902719 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:27.903079 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.903617 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.904511 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.904776 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.904867 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.905672 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:27.905675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:27.905677 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:27.905691 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:27.905712 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:27.905720 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:27.905780 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.905839 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.905892 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260811 22:24:27.909106 28363 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:24:27.912441 28363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:24:27.927275 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.921062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:39847 (local address 127.0.0.1:41212)
0811 22:24:27.921528 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.921583 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.921734 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.922807 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.922814 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.922838 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:27.922978 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.922984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.924343 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.924346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.925714 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.925725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.926717 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.926721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.926825 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.926861 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.926984 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.927124 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":70,"threads_started":1}
I20260811 22:24:27.927495 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.921196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.29.3.67:39847)
0811 22:24:27.921982 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.921994 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.922019 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:24:27.922096 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.922102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.922368 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.922623 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:27.923080 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.924223 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.925855 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.926594 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.926703 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.926913 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.927012 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.927115 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":202,"threads_started":1}
I20260811 22:24:27.927626 29146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.921090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:33323 (local address 127.0.0.1:44612)
0811 22:24:27.922877 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.922890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.922984 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.923568 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.923571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.923589 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:27.923695 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.923701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.924979 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.924983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.926100 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.926121 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.927188 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.927192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.927283 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.927346 (+ 63us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.927411 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.927480 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1730,"mutex_wait_us":19,"thread_start_us":113,"threads_started":1}
I20260811 22:24:27.928067 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.922112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.29.3.65:41841)
0811 22:24:27.922528 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.922533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.923276 (+ 743us) server_negotiation.cc:407] Connection header received
0811 22:24:27.923363 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.923367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.923526 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.923672 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:27.923890 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.924783 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.926051 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.926882 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.926989 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.927189 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.927797 (+ 608us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.927908 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":129,"threads_started":1}
I20260811 22:24:27.928393 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.921762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44612 (local address 127.29.3.66:33323)
0811 22:24:27.922745 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.922756 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.922992 (+ 236us) server_negotiation.cc:407] Connection header received
0811 22:24:27.923209 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.923215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.923404 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.923844 (+ 440us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:27.923980 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.924814 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.926230 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.927042 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.927887 (+ 845us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.928075 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.928141 (+ 66us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.928240 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":109,"threads_started":1}
I20260811 22:24:27.928388 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.921920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:41841 (local address 127.0.0.1:40196)
0811 22:24:27.923168 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.923180 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.923268 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.923669 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.923672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.923689 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:27.923785 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.923792 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.924905 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.924908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.925944 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.925955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.927241 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.927244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.927465 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.927487 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.927544 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.928264 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1190,"mutex_wait_us":163,"thread_start_us":98,"threads_started":1}
I20260811 22:24:27.930486 29009 tablet_service.cc:1467] Processing CreateTablet for tablet 7045486dd944467587d6dbabed7181e7 (DEFAULT_TABLE table=test-table [id=dba903c24b5940d0a382b89648f8ab0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:27.930982 28806 tablet_service.cc:1467] Processing CreateTablet for tablet 7045486dd944467587d6dbabed7181e7 (DEFAULT_TABLE table=test-table [id=dba903c24b5940d0a382b89648f8ab0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:27.931617 29009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7045486dd944467587d6dbabed7181e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:27.931914 28806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7045486dd944467587d6dbabed7181e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:27.933012 28577 tablet_service.cc:1467] Processing CreateTablet for tablet 7045486dd944467587d6dbabed7181e7 (DEFAULT_TABLE table=test-table [id=dba903c24b5940d0a382b89648f8ab0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:27.934054 28577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7045486dd944467587d6dbabed7181e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:27.938359 29155 tablet_bootstrap.cc:492] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: Bootstrap starting.
I20260811 22:24:27.940580 29155 tablet_bootstrap.cc:654] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:27.941322 29155 log.cc:824] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:27.941581 29158 tablet_bootstrap.cc:492] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: Bootstrap starting.
I20260811 22:24:27.942167 29157 tablet_bootstrap.cc:492] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Bootstrap starting.
I20260811 22:24:27.943652 29155 tablet_bootstrap.cc:492] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: No bootstrap required, opened a new log
I20260811 22:24:27.943862 29155 ts_tablet_manager.cc:1397] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:24:27.943902 29158 tablet_bootstrap.cc:654] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:27.944375 29157 tablet_bootstrap.cc:654] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:27.944793 29158 log.cc:824] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:27.945187 29157 log.cc:824] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:27.946838 29158 tablet_bootstrap.cc:492] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: No bootstrap required, opened a new log
I20260811 22:24:27.947036 29158 ts_tablet_manager.cc:1397] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:24:27.947650 29155 raft_consensus.cc:348] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.947952 29155 raft_consensus.cc:374] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:27.948014 29155 raft_consensus.cc:729] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbd7babd74924d02a91c40ec1b7ec522, State: Initialized, Role: FOLLOWER
I20260811 22:24:27.948100 29157 tablet_bootstrap.cc:492] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: No bootstrap required, opened a new log
I20260811 22:24:27.948295 29157 ts_tablet_manager.cc:1397] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:24:27.948560 29155 consensus_queue.cc:260] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.949391 28906 heartbeater.cc:503] Master 127.29.3.126:44661 was elected leader, sending a full tablet report...
I20260811 22:24:27.949728 29155 ts_tablet_manager.cc:1428] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:24:27.951239 29158 raft_consensus.cc:348] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.951481 29158 raft_consensus.cc:374] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:27.951547 29158 raft_consensus.cc:729] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c396acc3455493c8c12bc6e86a37617, State: Initialized, Role: FOLLOWER
I20260811 22:24:27.951994 29158 consensus_queue.cc:260] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.952672 28692 heartbeater.cc:503] Master 127.29.3.126:44661 was elected leader, sending a full tablet report...
I20260811 22:24:27.953765 29157 raft_consensus.cc:348] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.954056 29157 raft_consensus.cc:374] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:27.954123 29157 raft_consensus.cc:729] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f89743b6b91417d985b8d0fb3ec8b09, State: Initialized, Role: FOLLOWER
I20260811 22:24:27.954149 29158 ts_tablet_manager.cc:1428] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260811 22:24:27.954676 29157 consensus_queue.cc:260] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.955451 29124 heartbeater.cc:503] Master 127.29.3.126:44661 was elected leader, sending a full tablet report...
I20260811 22:24:27.955683 29157 ts_tablet_manager.cc:1428] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260811 22:24:27.966473 28694 tablet.cc:2369] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:24:27.968927 29167 raft_consensus.cc:482] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:27.969180 29167 raft_consensus.cc:504] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.970427 29167 leader_election.cc:288] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbd7babd74924d02a91c40ec1b7ec522 (127.29.3.66:33323), 3c396acc3455493c8c12bc6e86a37617 (127.29.3.65:41841)
I20260811 22:24:27.974105 29132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.970652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:41841 (local address 127.29.3.67:46363)
0811 22:24:27.970807 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.970826 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.970945 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.971230 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.971232 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.971252 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:27.971387 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.971395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.972178 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.972182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.972996 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.973006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.973662 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.973666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.973776 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.973798 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.973874 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.973954 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 22:24:27.974109 29154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.970853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:46363 (local address 127.29.3.65:41841)
0811 22:24:27.971001 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.971006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.971022 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:24:27.971083 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.971085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.971129 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.971203 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:27.971513 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.972055 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.973142 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.973537 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.973644 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.973715 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.973903 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.973995 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 22:24:27.974404 29150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.970844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:54107 (local address 127.29.3.66:33323)
0811 22:24:27.970979 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.970984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.971361 (+ 377us) server_negotiation.cc:407] Connection header received
0811 22:24:27.971479 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.971483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.971533 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.971595 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:27.971899 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.972401 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.973403 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.973816 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.973908 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.973967 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.974151 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.974237 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:24:27.974838 28608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7045486dd944467587d6dbabed7181e7" candidate_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c396acc3455493c8c12bc6e86a37617" is_pre_election: true
I20260811 22:24:27.974470 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.970652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:33323 (local address 127.29.3.67:54107)
0811 22:24:27.971261 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:27.971273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:27.971377 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:27.971617 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:27.971620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:27.971636 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:27.971785 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.971791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.972507 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.972510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.973282 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:27.973291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.973941 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.973944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:27.974040 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.974057 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 22:24:27.974119 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 22:24:27.974186 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":241,"thread_start_us":136,"threads_started":1}
I20260811 22:24:27.975191 28608 raft_consensus.cc:2454] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f89743b6b91417d985b8d0fb3ec8b09 in term 0.
I20260811 22:24:27.975754 28937 leader_election.cc:302] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c396acc3455493c8c12bc6e86a37617, 8f89743b6b91417d985b8d0fb3ec8b09; no voters:
I20260811 22:24:27.975741 28834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7045486dd944467587d6dbabed7181e7" candidate_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd7babd74924d02a91c40ec1b7ec522" is_pre_election: true
I20260811 22:24:27.976061 28834 raft_consensus.cc:2454] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f89743b6b91417d985b8d0fb3ec8b09 in term 0.
I20260811 22:24:27.976068 29167 raft_consensus.cc:2781] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:27.976167 29167 raft_consensus.cc:482] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:27.976231 29167 raft_consensus.cc:3037] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:27.977356 29167 raft_consensus.cc:504] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.977715 29167 leader_election.cc:288] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [CANDIDATE]: Term 1 election: Requested vote from peers cbd7babd74924d02a91c40ec1b7ec522 (127.29.3.66:33323), 3c396acc3455493c8c12bc6e86a37617 (127.29.3.65:41841)
I20260811 22:24:27.978036 28834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7045486dd944467587d6dbabed7181e7" candidate_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbd7babd74924d02a91c40ec1b7ec522"
I20260811 22:24:27.978092 28608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7045486dd944467587d6dbabed7181e7" candidate_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c396acc3455493c8c12bc6e86a37617"
I20260811 22:24:27.978166 28834 raft_consensus.cc:3037] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:27.978219 28608 raft_consensus.cc:3037] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:27.979472 28834 raft_consensus.cc:2454] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f89743b6b91417d985b8d0fb3ec8b09 in term 1.
I20260811 22:24:27.979472 28608 raft_consensus.cc:2454] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f89743b6b91417d985b8d0fb3ec8b09 in term 1.
I20260811 22:24:27.979801 28934 leader_election.cc:302] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f89743b6b91417d985b8d0fb3ec8b09, cbd7babd74924d02a91c40ec1b7ec522; no voters:
I20260811 22:24:27.979984 29167 raft_consensus.cc:2781] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:27.980211 29167 raft_consensus.cc:686] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 1 LEADER]: Becoming Leader. State: Replica: 8f89743b6b91417d985b8d0fb3ec8b09, State: Running, Role: LEADER
I20260811 22:24:27.980571 29167 consensus_queue.cc:237] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } }
I20260811 22:24:27.984022 28363 catalog_manager.cc:5495] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f89743b6b91417d985b8d0fb3ec8b09 (127.29.3.67). New cstate: current_term: 1 leader_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f89743b6b91417d985b8d0fb3ec8b09" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 } health_report { overall_health: UNKNOWN } } }
I20260811 22:24:27.997392 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:27.993813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41224 (local address 127.29.3.67:39847)
0811 22:24:27.993968 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:27.993973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:27.993987 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:24:27.994089 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:27.994091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:27.994147 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:27.994243 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:24:27.994526 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.995048 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.995933 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:27.996164 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:27.996254 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:27.996546 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:24:27.996984 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:24:27.997109 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:27.997168 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 22:24:27.997223 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 22:24:28.004293 28834 raft_consensus.cc:1261] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 1 FOLLOWER]: Refusing update from remote peer 8f89743b6b91417d985b8d0fb3ec8b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:28.004293 28608 raft_consensus.cc:1261] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 1 FOLLOWER]: Refusing update from remote peer 8f89743b6b91417d985b8d0fb3ec8b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:28.005090 29167 consensus_queue.cc:1035] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c396acc3455493c8c12bc6e86a37617" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:28.005335 29172 consensus_queue.cc:1035] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbd7babd74924d02a91c40ec1b7ec522" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:28.012770 29178 mvcc.cc:205] Tried to move back new op lower bound from 7317451030536650752 to 7317451030451675136. Current Snapshot: MvccSnapshot[applied={T|T < 7317451030536650752}]
I20260811 22:24:28.014834 29179 mvcc.cc:205] Tried to move back new op lower bound from 7317451030536650752 to 7317451030451675136. Current Snapshot: MvccSnapshot[applied={T|T < 7317451030536650752}]
I20260811 22:24:28.016577 29180 mvcc.cc:205] Tried to move back new op lower bound from 7317451030536650752 to 7317451030451675136. Current Snapshot: MvccSnapshot[applied={T|T < 7317451030536650752 or (T in {7317451030536650752})}]
I20260811 22:24:28.022234 28363 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:28.022449 28363 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:28.024930 28363 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec19341f3efb421f9788b64e459d546e: Sending DeleteTablet for 3 replicas of tablet 7045486dd944467587d6dbabed7181e7
I20260811 22:24:28.025717 29009 tablet_service.cc:1514] Processing DeleteTablet for tablet 7045486dd944467587d6dbabed7181e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:28 UTC) from {username='slave'} at 127.0.0.1:41212
I20260811 22:24:28.025739 28806 tablet_service.cc:1514] Processing DeleteTablet for tablet 7045486dd944467587d6dbabed7181e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:28 UTC) from {username='slave'} at 127.0.0.1:44612
I20260811 22:24:28.025905 28577 tablet_service.cc:1514] Processing DeleteTablet for tablet 7045486dd944467587d6dbabed7181e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:28 UTC) from {username='slave'} at 127.0.0.1:40196
I20260811 22:24:28.026454 29193 tablet_replica.cc:330] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617: stopping tablet replica
I20260811 22:24:28.026566 29192 tablet_replica.cc:330] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: stopping tablet replica
I20260811 22:24:28.026897 29192 raft_consensus.cc:2229] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:24:28.027065 29193 raft_consensus.cc:2229] T 7045486dd944467587d6dbabed7181e7 P 3c396acc3455493c8c12bc6e86a37617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:28.027122 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 28466
I20260811 22:24:28.068449 29192 raft_consensus.cc:2258] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:28.069051 29191 tablet_replica.cc:330] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: stopping tablet replica
I20260811 22:24:28.069446 29191 raft_consensus.cc:2229] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:28.069809 29191 raft_consensus.cc:2258] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:28.071578 29192 ts_tablet_manager.cc:1905] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:28.073997 29192 ts_tablet_manager.cc:1918] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:28.074106 29192 log.cc:1196] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/7045486dd944467587d6dbabed7181e7
I20260811 22:24:28.074499 29192 ts_tablet_manager.cc:1939] T 7045486dd944467587d6dbabed7181e7 P 8f89743b6b91417d985b8d0fb3ec8b09: Deleting consensus metadata
I20260811 22:24:28.075522 29191 ts_tablet_manager.cc:1905] T 7045486dd944467587d6dbabed7181e7 P cbd7babd74924d02a91c40ec1b7ec522: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:28.075940 28344 catalog_manager.cc:4841] TS 8f89743b6b91417d985b8d0fb3ec8b09 (127.29.3.67:39847): tablet 7045486dd944467587d6dbabed7181e7 (table test-table [id=dba903c24b5940d0a382b89648f8ab0a]) successfully deleted
W20260811 22:24:28.076102 28342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:41841 (error 108)
I20260811 22:24:28.077226 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:28.076882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:41841 (local address 127.0.0.1:40200)
0811 22:24:28.077029 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:28.077113 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:41841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:24:28.077302 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 28696
W20260811 22:24:28.105592 28342 catalog_manager.cc:4584] TS 3c396acc3455493c8c12bc6e86a37617 (127.29.3.65:41841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7045486dd944467587d6dbabed7181e7: Network error: Client connection negotiation failed: client connection to 127.29.3.65:41841: connect: Connection refused (error 111)
I20260811 22:24:28.115886 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 28910
I20260811 22:24:28.126502 29152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:28.119234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:33323 (local address 127.0.0.1:44628)
0811 22:24:28.126215 (+ 6981us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:28.126386 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.66:33323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6833}
I20260811 22:24:28.138064 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 28315
2026-08-11T22:24:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:28 dist-test-slave-4tdj krb5kdc[29201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:28 dist-test-slave-4tdj krb5kdc[29201](info): set up 2 sockets
Aug 11 22:24:28 dist-test-slave-4tdj krb5kdc[29201](info): commencing operation
krb5kdc: starting...
W20260811 22:24:29.360973 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:29 dist-test-slave-4tdj krb5kdc[29201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:24:29.493841 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:41799
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:41799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:29.613292 29219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:29.613593 29219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:29.613655 29219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:29.617750 29219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:29.617826 29219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:29.617853 29219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:29.617874 29219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:29.617895 29219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:29.623484 29219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:41799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:29.624632 29219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:29.625706 29219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:29.634810 29225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:29.634810 29226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:29.634871 29229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:29.635001 29219 server_base.cc:1034] running on GCE node
I20260811 22:24:29.635730 29219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:29.636904 29219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:29.638074 29219 hybrid_clock.cc:648] HybridClock initialized: now 1786487069638054 us; error 34 us; skew 500 ppm
Aug 11 22:24:29 dist-test-slave-4tdj krb5kdc[29201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487069, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:29.645542 29219 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:29.646673 29219 webserver.cc:460] Webserver started at http://127.29.3.126:41051/ using document root <none> and password file <none>
I20260811 22:24:29.647279 29219 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:29.647349 29219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:29.647554 29219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:29.649742 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "a3fdee03868d49b68b880ab7a6857a5a"
format_stamp: "Formatted at 2026-08-11 22:24:29 on dist-test-slave-4tdj"
server_key: "90b86c4d4146177da8626b54a6e185e8"
server_key_iv: "e3b03182cf1068d802f4f5193981ad6b"
server_key_version: "encryptionkey@0"
I20260811 22:24:29.650377 29219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "a3fdee03868d49b68b880ab7a6857a5a"
format_stamp: "Formatted at 2026-08-11 22:24:29 on dist-test-slave-4tdj"
server_key: "90b86c4d4146177da8626b54a6e185e8"
server_key_iv: "e3b03182cf1068d802f4f5193981ad6b"
server_key_version: "encryptionkey@0"
I20260811 22:24:29.654596 29219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:24:29.657281 29239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:29.658707 29219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:29.658835 29219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "a3fdee03868d49b68b880ab7a6857a5a"
format_stamp: "Formatted at 2026-08-11 22:24:29 on dist-test-slave-4tdj"
server_key: "90b86c4d4146177da8626b54a6e185e8"
server_key_iv: "e3b03182cf1068d802f4f5193981ad6b"
server_key_version: "encryptionkey@0"
I20260811 22:24:29.658944 29219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:29.669770 29219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:29.672976 29219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:29.673183 29219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:29.680554 29219 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:41799
I20260811 22:24:29.680595 29326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:41799 every 8 connection(s)
I20260811 22:24:29.681550 29219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:29.683810 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 29219
I20260811 22:24:29.683948 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:29.684262 29709 external_mini_cluster.cc:1428] Setting key ba9246676b6c3d578248417e8ccbafc2
I20260811 22:24:29.684911 29327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:29.691673 29327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a: Bootstrap starting.
Aug 11 22:24:29 dist-test-slave-4tdj krb5kdc[29201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:29.694540 29327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:29.695426 29327 log.cc:824] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:29.697834 29327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a: No bootstrap required, opened a new log
I20260811 22:24:29.699025 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:29.685677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40762 (local address 127.29.3.126:41799)
0811 22:24:29.686089 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:29.686105 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:29.686145 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:24:29.686848 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:29.686867 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:29.687200 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:29.687488 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:29.688380 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:29.689409 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:29.690632 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:29.691099 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:29.691216 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:29.694138 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:29.694144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:29.694159 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:29.694191 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:29.696052 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:29.696589 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:29.696595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:29.696600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:29.696721 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:29.696983 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:29.696985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:29.696986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:29.697412 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:29.697565 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:29.697872 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 22:24:29.698195 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260811 22:24:29.701326 29327 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fdee03868d49b68b880ab7a6857a5a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41799 } }
I20260811 22:24:29.701615 29327 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:29.701674 29327 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3fdee03868d49b68b880ab7a6857a5a, State: Initialized, Role: FOLLOWER
I20260811 22:24:29.702090 29327 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fdee03868d49b68b880ab7a6857a5a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41799 } }
I20260811 22:24:29.702219 29327 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:29.702282 29327 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:29.702404 29327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:29.703379 29327 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fdee03868d49b68b880ab7a6857a5a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41799 } }
I20260811 22:24:29.703727 29327 leader_election.cc:302] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3fdee03868d49b68b880ab7a6857a5a; no voters:
I20260811 22:24:29.704016 29327 leader_election.cc:288] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:29.704121 29333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:29.704362 29333 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [term 1 LEADER]: Becoming Leader. State: Replica: a3fdee03868d49b68b880ab7a6857a5a, State: Running, Role: LEADER
I20260811 22:24:29.704681 29333 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fdee03868d49b68b880ab7a6857a5a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41799 } }
I20260811 22:24:29.705245 29327 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:29.706431 29339 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3fdee03868d49b68b880ab7a6857a5a. Latest consensus state: current_term: 1 leader_uuid: "a3fdee03868d49b68b880ab7a6857a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fdee03868d49b68b880ab7a6857a5a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41799 } } }
I20260811 22:24:29.706588 29339 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:29.706815 29337 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3fdee03868d49b68b880ab7a6857a5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3fdee03868d49b68b880ab7a6857a5a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 41799 } } }
I20260811 22:24:29.706918 29337 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:29.707048 29349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:29.710954 29349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:29.716864 29349 catalog_manager.cc:1347] Generated new cluster ID: 0da19a1facc743ba8370dfa75812eef8
I20260811 22:24:29.716962 29349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:29.736403 29349 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:29.737450 29349 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:29.744737 29349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a: Generated new TSK 0
I20260811 22:24:29.745507 29349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:24:29.775290 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41799
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:29.894023 29373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:29.894359 29373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:29.894444 29373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:29.898623 29373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:29.898702 29373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:29.898764 29373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:29.904410 29373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:29.905679 29373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:29.906858 29373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 22:24:29.917469 29373 server_base.cc:1034] running on GCE node
W20260811 22:24:29.917423 29379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:29.917438 29381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:29.917675 29383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:29.918116 29373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:29.918898 29373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:29.920079 29373 hybrid_clock.cc:648] HybridClock initialized: now 1786487069920059 us; error 36 us; skew 500 ppm
Aug 11 22:24:29 dist-test-slave-4tdj krb5kdc[29201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487069, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:29.923193 29373 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:24:29.924217 29373 webserver.cc:460] Webserver started at http://127.29.3.65:33933/ using document root <none> and password file <none>
I20260811 22:24:29.924785 29373 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:29.924850 29373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:29.925055 29373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:29.927343 29373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "b07c5e87b5414ad6b0ddbffa11af682d"
format_stamp: "Formatted at 2026-08-11 22:24:29 on dist-test-slave-4tdj"
server_key: "695a8a22a8f8414239184651f434699c"
server_key_iv: "7c652c0a86347a08543f82fb11511f02"
server_key_version: "encryptionkey@0"
I20260811 22:24:29.927911 29373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "b07c5e87b5414ad6b0ddbffa11af682d"
format_stamp: "Formatted at 2026-08-11 22:24:29 on dist-test-slave-4tdj"
server_key: "695a8a22a8f8414239184651f434699c"
server_key_iv: "7c652c0a86347a08543f82fb11511f02"
server_key_version: "encryptionkey@0"
I20260811 22:24:29.931826 29373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:24:29.934585 29395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:29.935907 29373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 22:24:29.936053 29373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "b07c5e87b5414ad6b0ddbffa11af682d"
format_stamp: "Formatted at 2026-08-11 22:24:29 on dist-test-slave-4tdj"
server_key: "695a8a22a8f8414239184651f434699c"
server_key_iv: "7c652c0a86347a08543f82fb11511f02"
server_key_version: "encryptionkey@0"
I20260811 22:24:29.936168 29373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:29.964188 29373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:29.967469 29373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:29.967679 29373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:29.968326 29373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:29.969260 29373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:29.969312 29373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:29.969372 29373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:29.969401 29373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:29.978780 29373 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:36963
I20260811 22:24:29.978848 29598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:36963 every 8 connection(s)
I20260811 22:24:29.979730 29373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:29.983325 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 29373
I20260811 22:24:29.983449 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:24:29.983745 29709 external_mini_cluster.cc:1428] Setting key 4370a00882d26b6813326c7bde1e43b6
Aug 11 22:24:29 dist-test-slave-4tdj krb5kdc[29201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487069, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:24:29.994983 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:29.981682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:38671 (local address 127.29.3.126:41799)
0811 22:24:29.981827 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:29.981832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:29.982710 (+ 878us) server_negotiation.cc:407] Connection header received
0811 22:24:29.984109 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:29.984113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:29.984173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:29.984259 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:29.985746 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:29.986489 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:29.987669 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:29.987902 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:29.987996 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:29.990667 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:29.990672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:29.990675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:29.990712 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:29.992525 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:29.993129 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:29.993135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:29.993136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:29.993237 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:29.993662 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:29.993666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:29.993668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:29.993937 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:29.994050 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:29.994716 (+ 666us) server_negotiation.cc:299] Negotiation successful
0811 22:24:29.994850 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:24:29.995937 29607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:29.982037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:41799 (local address 127.29.3.65:38671)
0811 22:24:29.982556 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:29.982601 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:29.983884 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:29.984457 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:29.984466 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:29.984964 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:29.985586 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:29.985602 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:29.986601 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:29.986604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:29.987548 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:29.987558 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:29.988072 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:29.988076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:29.988250 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:29.988835 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:29.988850 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:29.990451 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:29.992696 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:29.992703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:29.992719 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:29.993020 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:29.993366 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:29.993374 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:29.993375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:29.993557 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:29.994045 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:29.994051 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:29.994407 (+ 356us) client_negotiation.cc:769] Sending connection context
0811 22:24:29.994644 (+ 237us) client_negotiation.cc:240] Negotiation successful
0811 22:24:29.994959 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:24:29.997429 29600 heartbeater.cc:347] Connected to a master server at 127.29.3.126:41799
I20260811 22:24:29.997725 29600 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:29.998283 29600 heartbeater.cc:511] Master 127.29.3.126:41799 requested a full tablet report, sending...
I20260811 22:24:30.000053 29268 ts_manager.cc:194] Registered new tserver with Master: b07c5e87b5414ad6b0ddbffa11af682d (127.29.3.65:36963)
I20260811 22:24:30.001448 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:38671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:24:30.017295 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41799
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:30.138777 29611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:30.139068 29611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:30.139113 29611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:30.143270 29611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:30.143348 29611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:30.143410 29611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:30.148939 29611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:30.150146 29611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:30.151418 29611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:30.161720 29617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:30.161754 29619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:30.161756 29622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:30.161900 29611 server_base.cc:1034] running on GCE node
I20260811 22:24:30.162393 29611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:30.163136 29611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:30.164327 29611 hybrid_clock.cc:648] HybridClock initialized: now 1786487070164302 us; error 36 us; skew 500 ppm
Aug 11 22:24:30 dist-test-slave-4tdj krb5kdc[29201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487070, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:30.167308 29611 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:24:30.168226 29611 webserver.cc:460] Webserver started at http://127.29.3.66:40183/ using document root <none> and password file <none>
I20260811 22:24:30.168788 29611 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:30.168848 29611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:30.169036 29611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:30.171254 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "d1320afd2dff4f8296b1a3abc1f916e2"
format_stamp: "Formatted at 2026-08-11 22:24:30 on dist-test-slave-4tdj"
server_key: "2dd70ba9b969348433fca8e41890b9ab"
server_key_iv: "ca6546a0ad50d763636911c0f036a308"
server_key_version: "encryptionkey@0"
I20260811 22:24:30.171792 29611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "d1320afd2dff4f8296b1a3abc1f916e2"
format_stamp: "Formatted at 2026-08-11 22:24:30 on dist-test-slave-4tdj"
server_key: "2dd70ba9b969348433fca8e41890b9ab"
server_key_iv: "ca6546a0ad50d763636911c0f036a308"
server_key_version: "encryptionkey@0"
I20260811 22:24:30.175750 29611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:24:30.178391 29638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:30.179741 29611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:30.179860 29611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "d1320afd2dff4f8296b1a3abc1f916e2"
format_stamp: "Formatted at 2026-08-11 22:24:30 on dist-test-slave-4tdj"
server_key: "2dd70ba9b969348433fca8e41890b9ab"
server_key_iv: "ca6546a0ad50d763636911c0f036a308"
server_key_version: "encryptionkey@0"
I20260811 22:24:30.179980 29611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:30.206207 29611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:30.209811 29611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:30.209986 29611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:30.210681 29611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:30.211575 29611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:30.211634 29611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:30.211678 29611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:30.211696 29611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:30.220422 29611 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:38045
I20260811 22:24:30.220445 29851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:38045 every 8 connection(s)
I20260811 22:24:30.221369 29611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:30.225690 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 29611
I20260811 22:24:30.225811 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:24:30.226130 29709 external_mini_cluster.cc:1428] Setting key 07fd218393431eae19d682ce32ba9381
Aug 11 22:24:30 dist-test-slave-4tdj krb5kdc[29201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487070, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:30.236603 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.223263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:41445 (local address 127.29.3.126:41799)
0811 22:24:30.223390 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.223393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.224288 (+ 895us) server_negotiation.cc:407] Connection header received
0811 22:24:30.225701 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.225704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.225770 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.225854 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:30.227381 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.227910 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.229102 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.229336 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.229429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.232299 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:30.232304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:30.232308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:30.232346 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:30.233953 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:30.234723 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:30.234726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:30.234727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:30.234816 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:30.235296 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:30.235300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:30.235301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:30.235517 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:30.235600 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.236349 (+ 749us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.236473 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:24:30.237828 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.223616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:41799 (local address 127.29.3.66:41445)
0811 22:24:30.224109 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.224155 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.225459 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.226100 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.226110 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.226676 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:30.227214 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.227230 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.228038 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.228041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.228969 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.228981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.229491 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.229495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.229677 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.230413 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:30.230431 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:30.232081 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:30.234127 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:30.234135 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:30.234158 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:30.234606 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:30.234913 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:30.234916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:30.234918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:30.235144 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:30.235627 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:30.235636 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:30.236024 (+ 388us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.236279 (+ 255us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.236685 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":123,"threads_started":1}
I20260811 22:24:30.239089 29852 heartbeater.cc:347] Connected to a master server at 127.29.3.126:41799
I20260811 22:24:30.239359 29852 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:30.239962 29852 heartbeater.cc:511] Master 127.29.3.126:41799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
I20260811 22:24:30.241082 29268 ts_manager.cc:194] Registered new tserver with Master: d1320afd2dff4f8296b1a3abc1f916e2 (127.29.3.66:38045)
I20260811 22:24:30.241762 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:41445
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:24:30.255329 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41799
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:30.373347 29863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:30.373643 29863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:30.373708 29863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:30.377854 29863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:30.377924 29863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:30.377986 29863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:30.383554 29863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:30.384711 29863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:30.385783 29863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:30.396528 29871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:30.396519 29872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:30.396528 29878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:30.396690 29863 server_base.cc:1034] running on GCE node
I20260811 22:24:30.397373 29863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:30.398239 29863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:30.399467 29863 hybrid_clock.cc:648] HybridClock initialized: now 1786487070399444 us; error 35 us; skew 500 ppm
Aug 11 22:24:30 dist-test-slave-4tdj krb5kdc[29201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487070, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:30.402489 29863 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:24:30.403491 29863 webserver.cc:460] Webserver started at http://127.29.3.67:34427/ using document root <none> and password file <none>
I20260811 22:24:30.404083 29863 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:30.404157 29863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:30.404389 29863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:30.406755 29863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "5134543f6e7f4614970bcb64180d5532"
format_stamp: "Formatted at 2026-08-11 22:24:30 on dist-test-slave-4tdj"
server_key: "e2a13591f7187fc28a77805e5b2c1f9e"
server_key_iv: "8abd0c74ef7bb9128362605f67eccf49"
server_key_version: "encryptionkey@0"
I20260811 22:24:30.407403 29863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "5134543f6e7f4614970bcb64180d5532"
format_stamp: "Formatted at 2026-08-11 22:24:30 on dist-test-slave-4tdj"
server_key: "e2a13591f7187fc28a77805e5b2c1f9e"
server_key_iv: "8abd0c74ef7bb9128362605f67eccf49"
server_key_version: "encryptionkey@0"
I20260811 22:24:30.411531 29863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:24:30.414409 29889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:30.415774 29863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:30.415901 29863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "5134543f6e7f4614970bcb64180d5532"
format_stamp: "Formatted at 2026-08-11 22:24:30 on dist-test-slave-4tdj"
server_key: "e2a13591f7187fc28a77805e5b2c1f9e"
server_key_iv: "8abd0c74ef7bb9128362605f67eccf49"
server_key_version: "encryptionkey@0"
I20260811 22:24:30.416018 29863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:30.428925 29863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:30.432479 29863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:30.432713 29863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:30.433346 29863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:30.434286 29863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:30.434376 29863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:30.434434 29863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:30.434458 29863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:30.443486 29863 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:36481
I20260811 22:24:30.443532 30100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:36481 every 8 connection(s)
I20260811 22:24:30.444427 29863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:30.453064 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 29863
I20260811 22:24:30.453203 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:24:30.453543 29709 external_mini_cluster.cc:1428] Setting key c88b1fbbdd3255e8a05daa74710635b4
Aug 11 22:24:30 dist-test-slave-4tdj krb5kdc[29201](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487070, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:30.458568 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.446358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:52741 (local address 127.29.3.126:41799)
0811 22:24:30.446524 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.446530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.447376 (+ 846us) server_negotiation.cc:407] Connection header received
0811 22:24:30.448662 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.448665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.448716 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.448788 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:30.449985 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.450581 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.451704 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.451907 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.451975 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.454551 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:30.454554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:30.454556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:30.454582 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:30.456520 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:30.457021 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:30.457024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:30.457026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:30.457107 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:30.457426 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:30.457430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:30.457431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:30.457624 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:30.457710 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.458244 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.458406 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:24:30.459429 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.446754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:41799 (local address 127.29.3.67:52741)
0811 22:24:30.447220 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.447264 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.448455 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.448944 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.448951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.449329 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:30.449845 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.449859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.450711 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.450714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.451612 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.451621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.452030 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.452033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.452174 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.452701 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:30.452718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:30.454376 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:30.456629 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:30.456636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:30.456653 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:30.456924 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:30.457187 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:30.457190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:30.457193 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:30.457339 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:30.457731 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:30.457739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:30.458001 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.458178 (+ 177us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.458560 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260811 22:24:30.460541 30103 heartbeater.cc:347] Connected to a master server at 127.29.3.126:41799
I20260811 22:24:30.460776 30103 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:30.461261 30103 heartbeater.cc:511] Master 127.29.3.126:41799 requested a full tablet report, sending...
I20260811 22:24:30.462236 29268 ts_manager.cc:194] Registered new tserver with Master: 5134543f6e7f4614970bcb64180d5532 (127.29.3.67:36481)
I20260811 22:24:30.462852 29268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:52741
I20260811 22:24:30.467692 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:24:30.477464 29331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.469656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.29.3.126:41799)
0811 22:24:30.469801 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.469806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.469985 (+ 179us) server_negotiation.cc:407] Connection header received
0811 22:24:30.470108 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.470111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.470160 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.470236 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:30.471018 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.471556 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.472463 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.472692 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.472782 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.474371 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:30.474375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:30.474377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:30.474408 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:30.475644 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:30.476154 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:30.476158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:30.476160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:30.476246 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:30.476546 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:30.476550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:30.476552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:30.476781 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:30.476922 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.477183 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.477292 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:24:30.480576 29268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:24:30.483777 29268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:24:30.498451 30123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.492449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:36481 (local address 127.0.0.1:59706)
0811 22:24:30.493575 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.493627 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.493761 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.494409 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.494411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.494430 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.494523 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.494530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.495663 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.495667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.496884 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.496896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.497863 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.497866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.497975 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.498016 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.498130 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.498250 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"mutex_wait_us":8,"thread_start_us":96,"threads_started":1}
I20260811 22:24:30.498461 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.493306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59706 (local address 127.29.3.67:36481)
0811 22:24:30.493980 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.493988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.494006 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:24:30.494071 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.494075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.494236 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.494403 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.494662 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.495545 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.497000 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.497737 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.497841 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.497996 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.498175 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.498287 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":168,"threads_started":1}
I20260811 22:24:30.498883 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.492498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38045 (local address 127.0.0.1:37362)
0811 22:24:30.493689 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.493700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.493787 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.494322 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.494330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.494356 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.494500 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.494508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.495950 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.495954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.497163 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.497175 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.498446 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.498451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.498537 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.498561 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.498671 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.498754 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":53,"thread_start_us":97,"threads_started":1}
I20260811 22:24:30.498944 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.492714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.29.3.66:38045)
0811 22:24:30.493326 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.493339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.493904 (+ 565us) server_negotiation.cc:407] Connection header received
0811 22:24:30.493977 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.493981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.494144 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.494266 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.494642 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.495796 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.497302 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.498112 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.498218 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.498382 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.498714 (+ 332us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.498813 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":110,"threads_started":1}
I20260811 22:24:30.499514 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.492699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:36963 (local address 127.0.0.1:56254)
0811 22:24:30.494575 (+ 1876us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.494588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.494677 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.495113 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.495116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.495134 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.495216 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.495222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.496578 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.496582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.497575 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.497584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.499152 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.499155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.499239 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.499260 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.499334 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.499397 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1807,"mutex_wait_us":22,"thread_start_us":122,"threads_started":1}
I20260811 22:24:30.502136 29730 tablet_service.cc:1467] Processing CreateTablet for tablet 1c24aa28a6f44ce7895086135fa4ebe9 (DEFAULT_TABLE table=test-table [id=fc3464c7268c4dc98ff716902b3f7407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:30.502137 29978 tablet_service.cc:1467] Processing CreateTablet for tablet 1c24aa28a6f44ce7895086135fa4ebe9 (DEFAULT_TABLE table=test-table [id=fc3464c7268c4dc98ff716902b3f7407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:30.503356 29730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c24aa28a6f44ce7895086135fa4ebe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:30.503366 29978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c24aa28a6f44ce7895086135fa4ebe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:30.506451 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.493288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.29.3.65:36963)
0811 22:24:30.493794 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.493810 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.494667 (+ 857us) server_negotiation.cc:407] Connection header received
0811 22:24:30.494790 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.494795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.494970 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.495115 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.495319 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.496430 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.497695 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.498495 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.498947 (+ 452us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.499085 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.499393 (+ 308us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.499498 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":91,"threads_started":1}
I20260811 22:24:30.509935 30138 tablet_bootstrap.cc:492] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Bootstrap starting.
I20260811 22:24:30.509810 29476 tablet_service.cc:1467] Processing CreateTablet for tablet 1c24aa28a6f44ce7895086135fa4ebe9 (DEFAULT_TABLE table=test-table [id=fc3464c7268c4dc98ff716902b3f7407]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:30.511089 29476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c24aa28a6f44ce7895086135fa4ebe9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:30.511839 30139 tablet_bootstrap.cc:492] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Bootstrap starting.
I20260811 22:24:30.512586 30138 tablet_bootstrap.cc:654] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:30.513955 30138 log.cc:824] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:30.514700 30139 tablet_bootstrap.cc:654] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:30.515586 30139 log.cc:824] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:30.516312 30138 tablet_bootstrap.cc:492] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: No bootstrap required, opened a new log
I20260811 22:24:30.516511 30138 ts_tablet_manager.cc:1397] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260811 22:24:30.517745 30139 tablet_bootstrap.cc:492] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: No bootstrap required, opened a new log
I20260811 22:24:30.517937 30139 ts_tablet_manager.cc:1397] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260811 22:24:30.520246 30138 raft_consensus.cc:348] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.520501 30138 raft_consensus.cc:374] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:30.520560 30138 raft_consensus.cc:729] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5134543f6e7f4614970bcb64180d5532, State: Initialized, Role: FOLLOWER
I20260811 22:24:30.521018 30138 consensus_queue.cc:260] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.521041 30139 raft_consensus.cc:348] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.521294 30139 raft_consensus.cc:374] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:30.521354 30139 raft_consensus.cc:729] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1320afd2dff4f8296b1a3abc1f916e2, State: Initialized, Role: FOLLOWER
I20260811 22:24:30.521704 30103 heartbeater.cc:503] Master 127.29.3.126:41799 was elected leader, sending a full tablet report...
I20260811 22:24:30.521762 30139 consensus_queue.cc:260] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.522451 29852 heartbeater.cc:503] Master 127.29.3.126:41799 was elected leader, sending a full tablet report...
I20260811 22:24:30.522852 30139 ts_tablet_manager.cc:1428] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:24:30.523149 30138 ts_tablet_manager.cc:1428] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260811 22:24:30.523593 30143 tablet_bootstrap.cc:492] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: Bootstrap starting.
I20260811 22:24:30.526221 30143 tablet_bootstrap.cc:654] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:30.527141 30143 log.cc:824] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:30.529036 30143 tablet_bootstrap.cc:492] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: No bootstrap required, opened a new log
I20260811 22:24:30.529233 30143 ts_tablet_manager.cc:1397] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:24:30.532691 30143 raft_consensus.cc:348] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.532923 30143 raft_consensus.cc:374] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:30.532976 30143 raft_consensus.cc:729] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07c5e87b5414ad6b0ddbffa11af682d, State: Initialized, Role: FOLLOWER
I20260811 22:24:30.533439 30143 consensus_queue.cc:260] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.534137 29600 heartbeater.cc:503] Master 127.29.3.126:41799 was elected leader, sending a full tablet report...
I20260811 22:24:30.534514 30143 ts_tablet_manager.cc:1428] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260811 22:24:30.695389 30106 tablet.cc:2369] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:24:30.722610 29853 tablet.cc:2369] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 22:24:30.731109 29606 tablet.cc:2369] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:24:30.868813 30147 raft_consensus.cc:482] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:30.869091 30147 raft_consensus.cc:504] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.869549 30150 raft_consensus.cc:482] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:30.869829 30150 raft_consensus.cc:504] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.870707 30147 leader_election.cc:288] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5134543f6e7f4614970bcb64180d5532 (127.29.3.67:36481), b07c5e87b5414ad6b0ddbffa11af682d (127.29.3.65:36963)
I20260811 22:24:30.871194 30150 leader_election.cc:288] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5134543f6e7f4614970bcb64180d5532 (127.29.3.67:36481), d1320afd2dff4f8296b1a3abc1f916e2 (127.29.3.66:38045)
I20260811 22:24:30.874991 30145 raft_consensus.cc:482] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:30.875226 30145 raft_consensus.cc:504] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.876520 30145 leader_election.cc:288] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1320afd2dff4f8296b1a3abc1f916e2 (127.29.3.66:38045), b07c5e87b5414ad6b0ddbffa11af682d (127.29.3.65:36963)
I20260811 22:24:30.876682 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.871263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:49157 (local address 127.29.3.67:36481)
0811 22:24:30.871407 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.871413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.871817 (+ 404us) server_negotiation.cc:407] Connection header received
0811 22:24:30.872023 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.872027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.872080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.872128 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.873054 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873800 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.875392 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.876035 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.876142 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.876217 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.876435 (+ 218us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.876521 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":64}
I20260811 22:24:30.876806 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.871506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:60831 (local address 127.29.3.65:36963)
0811 22:24:30.871608 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.871613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.871628 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:24:30.871685 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.871688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.871751 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.871833 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.872753 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873557 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.875560 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.876116 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.876181 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.876245 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.876597 (+ 352us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.876677 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 22:24:30.877101 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.870951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:36481 (local address 127.29.3.66:49157)
0811 22:24:30.871721 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.871736 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.871896 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.872800 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.872803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.872820 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.872924 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.872930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873917 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.873920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.875247 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.875261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.876172 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.876176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.876269 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.876308 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.876555 (+ 247us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.876938 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":68,"thread_start_us":64,"threads_started":1}
I20260811 22:24:30.877272 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.871482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:36481 (local address 127.29.3.65:40761)
0811 22:24:30.872047 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.872060 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.872147 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.872648 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.872651 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.872669 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.872801 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.872807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873861 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.873864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.875160 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.875174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.876863 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.876867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.876965 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.876992 (+ 27us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.877062 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.877136 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":22,"thread_start_us":141,"threads_started":1}
I20260811 22:24:30.877714 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.871836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:40761 (local address 127.29.3.67:36481)
0811 22:24:30.872442 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.872449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.872460 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:24:30.872497 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.872499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.872531 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.872606 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.872936 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873747 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.875305 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.876649 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.876751 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.876804 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.877476 (+ 672us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.877571 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":529,"thread_start_us":105,"threads_started":1}
I20260811 22:24:30.878104 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5134543f6e7f4614970bcb64180d5532" is_pre_election: true
I20260811 22:24:30.878371 30014 raft_consensus.cc:2454] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1320afd2dff4f8296b1a3abc1f916e2 in term 0.
I20260811 22:24:30.878594 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.870951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:36963 (local address 127.29.3.66:60831)
0811 22:24:30.871347 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.871363 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.871474 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.872545 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.872547 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.872559 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.872655 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.872663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873705 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.873709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.875342 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.875437 (+ 95us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.876338 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.876342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.876465 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.876487 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.878265 (+ 1778us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.878371 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":37,"thread_start_us":104,"threads_started":1}
I20260811 22:24:30.878922 29643 leader_election.cc:302] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5134543f6e7f4614970bcb64180d5532, d1320afd2dff4f8296b1a3abc1f916e2; no voters:
I20260811 22:24:30.879171 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5134543f6e7f4614970bcb64180d5532" is_pre_election: true
I20260811 22:24:30.879292 30014 raft_consensus.cc:2454] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b07c5e87b5414ad6b0ddbffa11af682d in term 0.
I20260811 22:24:30.879302 29518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" is_pre_election: true
I20260811 22:24:30.879513 29518 raft_consensus.cc:2454] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1320afd2dff4f8296b1a3abc1f916e2 in term 0.
I20260811 22:24:30.879681 29401 leader_election.cc:302] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5134543f6e7f4614970bcb64180d5532, b07c5e87b5414ad6b0ddbffa11af682d; no voters:
I20260811 22:24:30.879824 30147 raft_consensus.cc:2781] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:30.879915 30147 raft_consensus.cc:482] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:30.879952 30147 raft_consensus.cc:3037] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:30.879977 30150 raft_consensus.cc:2781] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:30.880048 30150 raft_consensus.cc:482] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:30.880084 30150 raft_consensus.cc:3037] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:30.881023 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.872150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:48993 (local address 127.29.3.66:38045)
0811 22:24:30.872295 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.872300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.872313 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:24:30.872353 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.872355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.872389 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.872448 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.872744 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873504 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.879758 (+ 6254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.880323 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.880406 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.880463 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.880821 (+ 358us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.880912 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 22:24:30.881378 30147 raft_consensus.cc:504] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.881426 30150 raft_consensus.cc:504] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.881791 30147 leader_election.cc:288] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [CANDIDATE]: Term 1 election: Requested vote from peers 5134543f6e7f4614970bcb64180d5532 (127.29.3.67:36481), b07c5e87b5414ad6b0ddbffa11af682d (127.29.3.65:36963)
I20260811 22:24:30.881789 30150 leader_election.cc:288] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [CANDIDATE]: Term 1 election: Requested vote from peers 5134543f6e7f4614970bcb64180d5532 (127.29.3.67:36481), d1320afd2dff4f8296b1a3abc1f916e2 (127.29.3.66:38045)
I20260811 22:24:30.881832 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.877379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:53615 (local address 127.29.3.65:36963)
0811 22:24:30.877790 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.877794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.877808 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:24:30.877849 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.877851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.878034 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.878111 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.878497 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.879138 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.880583 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.881172 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.881257 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.881310 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.881649 (+ 339us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.881720 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":333}
I20260811 22:24:30.882189 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.877146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:36963 (local address 127.29.3.67:53615)
0811 22:24:30.877626 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.877636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.877701 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.878188 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.878191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.878209 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.878363 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.878370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.879278 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.879281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.880405 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.880416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.881407 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.881411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.881514 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.881537 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.881948 (+ 411us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.882037 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":154,"threads_started":1}
I20260811 22:24:30.882530 30014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5134543f6e7f4614970bcb64180d5532"
I20260811 22:24:30.882649 30014 raft_consensus.cc:3037] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:30.882710 30013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5134543f6e7f4614970bcb64180d5532"
I20260811 22:24:30.883183 29518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07c5e87b5414ad6b0ddbffa11af682d"
I20260811 22:24:30.883359 29518 raft_consensus.cc:2379] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1320afd2dff4f8296b1a3abc1f916e2 in current term 1: Already voted for candidate b07c5e87b5414ad6b0ddbffa11af682d in this term.
I20260811 22:24:30.883397 29515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "5134543f6e7f4614970bcb64180d5532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" is_pre_election: true
I20260811 22:24:30.883526 29515 raft_consensus.cc:2379] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5134543f6e7f4614970bcb64180d5532 in current term 1: Already voted for candidate b07c5e87b5414ad6b0ddbffa11af682d in this term.
I20260811 22:24:30.884444 30014 raft_consensus.cc:2454] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1320afd2dff4f8296b1a3abc1f916e2 in term 1.
I20260811 22:24:30.884716 29643 leader_election.cc:302] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5134543f6e7f4614970bcb64180d5532, d1320afd2dff4f8296b1a3abc1f916e2; no voters: b07c5e87b5414ad6b0ddbffa11af682d
I20260811 22:24:30.884758 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.871732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38045 (local address 127.29.3.65:48993)
0811 22:24:30.872150 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.872163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.872263 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.872474 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.872477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.872491 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.872613 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.872621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.873640 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.873643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.879574 (+ 5931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.879584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.880554 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.880558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.880683 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.880707 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.884519 (+ 3812us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.884607 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":32,"thread_start_us":111,"threads_started":1}
I20260811 22:24:30.884893 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.876871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38045 (local address 127.29.3.67:45411)
0811 22:24:30.877552 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:30.877570 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:30.877654 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:30.881513 (+ 3859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:30.881517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:30.881537 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:30.881637 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.881643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.882658 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.882661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.883721 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:30.883733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.884485 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.884488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:30.884593 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.884617 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:24:30.884699 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 22:24:30.884772 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602}
I20260811 22:24:30.885253 30147 raft_consensus.cc:2781] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:30.885501 30147 raft_consensus.cc:686] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 1 LEADER]: Becoming Leader. State: Replica: d1320afd2dff4f8296b1a3abc1f916e2, State: Running, Role: LEADER
I20260811 22:24:30.885586 29768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" is_pre_election: true
I20260811 22:24:30.885869 30147 consensus_queue.cc:237] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } }
I20260811 22:24:30.886466 29764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1320afd2dff4f8296b1a3abc1f916e2"
I20260811 22:24:30.886833 29398 leader_election.cc:302] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b07c5e87b5414ad6b0ddbffa11af682d; no voters: 5134543f6e7f4614970bcb64180d5532, d1320afd2dff4f8296b1a3abc1f916e2
I20260811 22:24:30.886926 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.878712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:45411 (local address 127.29.3.66:38045)
0811 22:24:30.881181 (+ 2469us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.881185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.881199 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:24:30.881236 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.881239 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.881284 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.881355 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:30.881822 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.882542 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.883891 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.884360 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.884444 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.884490 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.886732 (+ 2242us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.886811 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2408,"thread_start_us":140,"threads_started":1}
I20260811 22:24:30.887068 30150 raft_consensus.cc:2726] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260811 22:24:30.887235 29764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c24aa28a6f44ce7895086135fa4ebe9" candidate_uuid: "5134543f6e7f4614970bcb64180d5532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" is_pre_election: true
I20260811 22:24:30.887604 29891 leader_election.cc:302] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5134543f6e7f4614970bcb64180d5532; no voters: b07c5e87b5414ad6b0ddbffa11af682d, d1320afd2dff4f8296b1a3abc1f916e2
I20260811 22:24:30.887903 30145 raft_consensus.cc:2726] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260811 22:24:30.889374 29268 catalog_manager.cc:5495] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1320afd2dff4f8296b1a3abc1f916e2 (127.29.3.66). New cstate: current_term: 1 leader_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1320afd2dff4f8296b1a3abc1f916e2" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 } health_report { overall_health: UNKNOWN } } }
I20260811 22:24:30.955546 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:30.952133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.29.3.66:38045)
0811 22:24:30.952260 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:30.952263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:30.952277 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:24:30.952355 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:30.952357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:30.952415 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:30.952504 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:24:30.952741 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.953248 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.954065 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:30.954276 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:30.954371 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:30.954726 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:24:30.955157 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:24:30.955268 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:30.955387 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 22:24:30.955424 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:24:30.961560 29518 raft_consensus.cc:1261] T 1c24aa28a6f44ce7895086135fa4ebe9 P b07c5e87b5414ad6b0ddbffa11af682d [term 1 FOLLOWER]: Refusing update from remote peer d1320afd2dff4f8296b1a3abc1f916e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:30.961560 30014 raft_consensus.cc:1261] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 1 FOLLOWER]: Refusing update from remote peer d1320afd2dff4f8296b1a3abc1f916e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:30.962159 30163 consensus_queue.cc:1035] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b07c5e87b5414ad6b0ddbffa11af682d" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 36963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:30.962381 30147 consensus_queue.cc:1035] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5134543f6e7f4614970bcb64180d5532" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:30.971529 30175 mvcc.cc:205] Tried to move back new op lower bound from 7317451042651299840 to 7317451042351570944. Current Snapshot: MvccSnapshot[applied={T|T < 7317451042651299840}]
I20260811 22:24:30.979504 29268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:30.979698 29268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:30.982424 29268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3fdee03868d49b68b880ab7a6857a5a: Sending DeleteTablet for 3 replicas of tablet 1c24aa28a6f44ce7895086135fa4ebe9
I20260811 22:24:30.983296 29978 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c24aa28a6f44ce7895086135fa4ebe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:30 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:59706
I20260811 22:24:30.983378 29476 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c24aa28a6f44ce7895086135fa4ebe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:30 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:56254
I20260811 22:24:30.984499 29730 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c24aa28a6f44ce7895086135fa4ebe9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:30 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:37362
I20260811 22:24:30.984587 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 29373
I20260811 22:24:30.987818 30187 tablet_replica.cc:330] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: stopping tablet replica
I20260811 22:24:31.004796 30187 raft_consensus.cc:2229] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:24:31.005535 30187 raft_consensus.cc:2258] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:31.007681 30187 ts_tablet_manager.cc:1905] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:31.011176 30186 tablet_replica.cc:330] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: stopping tablet replica
I20260811 22:24:31.011597 30187 ts_tablet_manager.cc:1918] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:31.011718 30187 log.cc:1196] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/1c24aa28a6f44ce7895086135fa4ebe9
I20260811 22:24:31.011656 30186 raft_consensus.cc:2229] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:31.012027 30187 ts_tablet_manager.cc:1939] T 1c24aa28a6f44ce7895086135fa4ebe9 P d1320afd2dff4f8296b1a3abc1f916e2: Deleting consensus metadata
W20260811 22:24:31.012182 29240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:36963 (error 108)
I20260811 22:24:31.012431 30186 raft_consensus.cc:2258] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:31.013074 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 29611
I20260811 22:24:31.019590 30186 ts_tablet_manager.cc:1905] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:31.023212 30186 ts_tablet_manager.cc:1918] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:31.023355 30186 log.cc:1196] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/1c24aa28a6f44ce7895086135fa4ebe9
I20260811 22:24:31.023757 30186 ts_tablet_manager.cc:1939] T 1c24aa28a6f44ce7895086135fa4ebe9 P 5134543f6e7f4614970bcb64180d5532: Deleting consensus metadata
I20260811 22:24:31.027100 29250 catalog_manager.cc:4841] TS 5134543f6e7f4614970bcb64180d5532 (127.29.3.67:36481): tablet 1c24aa28a6f44ce7895086135fa4ebe9 (table test-table [id=fc3464c7268c4dc98ff716902b3f7407]) successfully deleted
I20260811 22:24:31.031271 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.027350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:36963 (local address 127.0.0.1:56256)
0811 22:24:31.031025 (+ 3675us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.031147 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:36963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3555,"thread_start_us":1334,"threads_started":1}
W20260811 22:24:31.031530 29240 catalog_manager.cc:4584] TS b07c5e87b5414ad6b0ddbffa11af682d (127.29.3.65:36963): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c24aa28a6f44ce7895086135fa4ebe9: Network error: Client connection negotiation failed: client connection to 127.29.3.65:36963: connect: Connection refused (error 111)
I20260811 22:24:31.037225 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 29863
I20260811 22:24:31.043318 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.037397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:38045 (local address 127.0.0.1:37380)
0811 22:24:31.042890 (+ 5493us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.043002 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.66:38045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5329}
I20260811 22:24:31.055114 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 29219
2026-08-11T22:24:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2916 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T22:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:31Z Disabled control of system clock
I20260811 22:24:31.096647 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35535
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:24:31.213084 30199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:31.213392 30199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:31.213428 30199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:31.217954 30199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:31.218055 30199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:31.218081 30199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:31.218106 30199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:31.218134 30199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:31.223549 30199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:31.224813 30199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:31.226370 30199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:31.235648 30216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:31.235651 30205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.235706 30199 server_base.cc:1034] running on GCE node
W20260811 22:24:31.235648 30206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.236521 30199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:31.237681 30199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:31.238837 30199 hybrid_clock.cc:648] HybridClock initialized: now 1786487071238812 us; error 39 us; skew 500 ppm
I20260811 22:24:31.241010 30199 webserver.cc:460] Webserver started at http://127.29.3.126:40153/ using document root <none> and password file <none>
I20260811 22:24:31.241603 30199 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:31.241665 30199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:31.241840 30199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:31.244099 30199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "289054f760e54f3a84b4581b40592760"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "a523aad0afa0e2d9dbd7b798efc988f7"
server_key_iv: "76efc7231a66974991211daa009ef191"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.244669 30199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "289054f760e54f3a84b4581b40592760"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "a523aad0afa0e2d9dbd7b798efc988f7"
server_key_iv: "76efc7231a66974991211daa009ef191"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.248657 30199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:31.251161 30224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.252424 30199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:24:31.252548 30199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "289054f760e54f3a84b4581b40592760"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "a523aad0afa0e2d9dbd7b798efc988f7"
server_key_iv: "76efc7231a66974991211daa009ef191"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.252650 30199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:31.264314 30199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:31.265056 30199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:31.265266 30199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:31.272575 30199 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:35535
I20260811 22:24:31.272645 30328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:35535 every 8 connection(s)
I20260811 22:24:31.273638 30199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:31.276991 30329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:31.283241 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760: Bootstrap starting.
I20260811 22:24:31.283416 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30199
I20260811 22:24:31.283550 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:31.283854 29709 external_mini_cluster.cc:1428] Setting key 8f0980fa858ac8f3f1fd9db2c5e3a2dd
I20260811 22:24:31.286264 30329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:31.287138 30329 log.cc:824] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:31.289129 30329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760: No bootstrap required, opened a new log
I20260811 22:24:31.292987 30329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289054f760e54f3a84b4581b40592760" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35535 } }
I20260811 22:24:31.293228 30329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:31.293265 30329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289054f760e54f3a84b4581b40592760, State: Initialized, Role: FOLLOWER
I20260811 22:24:31.293643 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.285354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.29.3.126:35535)
0811 22:24:31.286085 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.286100 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.286139 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:24:31.286947 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.286970 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.287464 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.287802 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:31.288270 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.289282 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.290192 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.290748 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.290902 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.291786 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:31.291792 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:31.291807 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:31.292216 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:31.292274 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:31.292295 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:31.292385 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.292535 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.292811 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":156,"threads_started":1}
I20260811 22:24:31.293735 30329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289054f760e54f3a84b4581b40592760" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35535 } }
I20260811 22:24:31.293936 30329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:31.293988 30329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:31.294077 30329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:31.295218 30329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289054f760e54f3a84b4581b40592760" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35535 } }
I20260811 22:24:31.295599 30329 leader_election.cc:302] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 289054f760e54f3a84b4581b40592760; no voters:
I20260811 22:24:31.295943 30329 leader_election.cc:288] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:31.296066 30340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:31.296312 30340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [term 1 LEADER]: Becoming Leader. State: Replica: 289054f760e54f3a84b4581b40592760, State: Running, Role: LEADER
I20260811 22:24:31.296617 30340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289054f760e54f3a84b4581b40592760" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35535 } }
I20260811 22:24:31.297154 30329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:31.298357 30347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 289054f760e54f3a84b4581b40592760. Latest consensus state: current_term: 1 leader_uuid: "289054f760e54f3a84b4581b40592760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289054f760e54f3a84b4581b40592760" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35535 } } }
I20260811 22:24:31.298485 30347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:31.298735 30342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "289054f760e54f3a84b4581b40592760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "289054f760e54f3a84b4581b40592760" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35535 } } }
I20260811 22:24:31.298835 30342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:31.298936 30358 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:31.302174 30358 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:31.307911 30358 catalog_manager.cc:1347] Generated new cluster ID: 4ffac8e103b14e8191f998b984545a03
I20260811 22:24:31.307996 30358 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:31.317447 30358 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:31.318526 30358 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:31.324857 30358 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760: Generated new TSK 0
I20260811 22:24:31.325762 30358 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:31.337129 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:35535
--builtin_ntp_servers=127.29.3.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:24:31.458278 30372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:31.458655 30372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:31.458717 30372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:31.462936 30372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:31.463016 30372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:31.463096 30372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:31.468225 30372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:31.469426 30372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:31.471040 30372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:31.481345 30385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:31.481354 30382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.481534 30372 server_base.cc:1034] running on GCE node
W20260811 22:24:31.481361 30383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.482069 30372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:31.482758 30372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:31.483923 30372 hybrid_clock.cc:648] HybridClock initialized: now 1786487071483903 us; error 29 us; skew 500 ppm
I20260811 22:24:31.485839 30372 webserver.cc:460] Webserver started at http://127.29.3.65:41359/ using document root <none> and password file <none>
I20260811 22:24:31.486450 30372 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:31.486524 30372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:31.486716 30372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:31.488828 30372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "f4853b24c5164eafa000ff5fb96bea25"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "44b45124f1f44351990b3fde9948865a"
server_key_iv: "8697128fd9d2e38a6a17a49aba3989d7"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.489362 30372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "f4853b24c5164eafa000ff5fb96bea25"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "44b45124f1f44351990b3fde9948865a"
server_key_iv: "8697128fd9d2e38a6a17a49aba3989d7"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.493178 30372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:24:31.495754 30396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.497087 30372 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:24:31.497208 30372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "f4853b24c5164eafa000ff5fb96bea25"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "44b45124f1f44351990b3fde9948865a"
server_key_iv: "8697128fd9d2e38a6a17a49aba3989d7"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.497339 30372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:31.518596 30372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:31.519354 30372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:31.519538 30372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:31.520181 30372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:31.521195 30372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:31.521257 30372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.521324 30372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:31.521349 30372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.530323 30372 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:45577
I20260811 22:24:31.530364 30627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:45577 every 8 connection(s)
I20260811 22:24:31.531292 30372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:31.535130 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30372
I20260811 22:24:31.535235 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:24:31.535490 29709 external_mini_cluster.cc:1428] Setting key 6e9e7b0edbde697bb32115f4b362ac70
I20260811 22:24:31.537561 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:35535
--builtin_ntp_servers=127.29.3.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:24:31.541270 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.533133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:60203 (local address 127.29.3.126:35535)
0811 22:24:31.533285 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.533289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.534395 (+ 1106us) server_negotiation.cc:407] Connection header received
0811 22:24:31.535914 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.535917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.535968 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.536056 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:31.537207 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.537978 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.539167 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.539471 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.539557 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.540810 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:31.540813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:31.540816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:31.540830 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:31.540854 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:31.540863 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:31.540928 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.541057 (+ 129us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.541121 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:24:31.542140 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.533578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35535 (local address 127.29.3.65:60203)
0811 22:24:31.534240 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.534282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:31.535693 (+ 1411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:31.536220 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:31.536228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:31.536338 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:31.537048 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.537067 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.538091 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.538093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.539042 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.539053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.539645 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.539649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.539853 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.540495 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:31.540497 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:31.540555 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:31.540645 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:31.540941 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:31.540947 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:31.540951 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:24:31.541042 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 22:24:31.541294 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260811 22:24:31.543428 30629 heartbeater.cc:347] Connected to a master server at 127.29.3.126:35535
I20260811 22:24:31.543670 30629 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:31.544220 30629 heartbeater.cc:511] Master 127.29.3.126:35535 requested a full tablet report, sending...
I20260811 22:24:31.545804 30248 ts_manager.cc:194] Registered new tserver with Master: f4853b24c5164eafa000ff5fb96bea25 (127.29.3.65:45577)
I20260811 22:24:31.546993 30248 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.65:60203
W20260811 22:24:31.658838 30634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:31.659152 30634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:31.659198 30634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:31.663379 30634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:31.663456 30634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:31.663519 30634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:31.668606 30634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:31.669925 30634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:31.671509 30634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:31.681800 30646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:31.681821 30642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.681923 30634 server_base.cc:1034] running on GCE node
W20260811 22:24:31.681821 30640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.682541 30634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:31.683269 30634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:31.684433 30634 hybrid_clock.cc:648] HybridClock initialized: now 1786487071684400 us; error 50 us; skew 500 ppm
I20260811 22:24:31.686367 30634 webserver.cc:460] Webserver started at http://127.29.3.66:45425/ using document root <none> and password file <none>
I20260811 22:24:31.686956 30634 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:31.687024 30634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:31.687233 30634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:31.689390 30634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "e744a4770d1f4cbdacb32b7a846809c5"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "e05f33950e1fd68799f98bb86c0bb036"
server_key_iv: "9c10133bfb609a3e711a4bf70bf0cf43"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.689949 30634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "e744a4770d1f4cbdacb32b7a846809c5"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "e05f33950e1fd68799f98bb86c0bb036"
server_key_iv: "9c10133bfb609a3e711a4bf70bf0cf43"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.693856 30634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:31.696584 30657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.698035 30634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:31.698156 30634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "e744a4770d1f4cbdacb32b7a846809c5"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "e05f33950e1fd68799f98bb86c0bb036"
server_key_iv: "9c10133bfb609a3e711a4bf70bf0cf43"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.698285 30634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:31.713080 30634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:31.713940 30634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:31.714141 30634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:31.714893 30634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:31.715930 30634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:31.715981 30634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.716039 30634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:31.716066 30634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.725014 30634 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:39801
I20260811 22:24:31.725041 30878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:39801 every 8 connection(s)
I20260811 22:24:31.725989 30634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:31.735176 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.727917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:58521 (local address 127.29.3.126:35535)
0811 22:24:31.728047 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.728050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.729186 (+ 1136us) server_negotiation.cc:407] Connection header received
0811 22:24:31.730597 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.730600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.730651 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.730730 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:31.731643 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.732193 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.733318 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.733547 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.733635 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.734749 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:31.734752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:31.734754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:31.734768 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:31.734789 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:31.734797 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:31.734853 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.734974 (+ 121us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.735030 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:24:31.735924 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30634
I20260811 22:24:31.736056 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:24:31.736291 30889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.728343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35535 (local address 127.29.3.66:58521)
0811 22:24:31.729020 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.729062 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:31.730404 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:31.730895 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:31.730902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:31.730989 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:31.731515 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.731530 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.732285 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.732287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.733163 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.733172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.733674 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.733677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.733828 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.734474 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:31.734476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:31.734520 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:31.734590 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:31.734859 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:31.734865 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:31.734869 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 22:24:31.734960 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 22:24:31.735209 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260811 22:24:31.736389 29709 external_mini_cluster.cc:1428] Setting key ca7519bf2435fcadb3d3a19246219a1c
I20260811 22:24:31.737588 30881 heartbeater.cc:347] Connected to a master server at 127.29.3.126:35535
I20260811 22:24:31.737834 30881 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:31.738528 30881 heartbeater.cc:511] Master 127.29.3.126:35535 requested a full tablet report, sending...
I20260811 22:24:31.738708 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:35535
--builtin_ntp_servers=127.29.3.84:42499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:24:31.739703 30259 ts_manager.cc:194] Registered new tserver with Master: e744a4770d1f4cbdacb32b7a846809c5 (127.29.3.66:39801)
I20260811 22:24:31.740302 30259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.66:58521
W20260811 22:24:31.855538 30890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:31.855816 30890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:31.855855 30890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:31.859993 30890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:31.860059 30890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:31.860116 30890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:31.864974 30890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:42499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:31.866106 30890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:31.867650 30890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:31.877741 30905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:31.877744 30906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:31.877744 30908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:31.878554 30890 server_base.cc:1034] running on GCE node
I20260811 22:24:31.878984 30890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:31.879655 30890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:31.880823 30890 hybrid_clock.cc:648] HybridClock initialized: now 1786487071880804 us; error 30 us; skew 500 ppm
I20260811 22:24:31.882790 30890 webserver.cc:460] Webserver started at http://127.29.3.67:45915/ using document root <none> and password file <none>
I20260811 22:24:31.883405 30890 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:31.883479 30890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:31.883697 30890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:31.886001 30890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "1861188064d64b27881f03e3cc21d710"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "467dc5dcd1367f19eb0deb87f4f7f657"
server_key_iv: "bea6b41a8713d32dc927b83b0be8eba2"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.886667 30890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "1861188064d64b27881f03e3cc21d710"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "467dc5dcd1367f19eb0deb87f4f7f657"
server_key_iv: "bea6b41a8713d32dc927b83b0be8eba2"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.890857 30890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:24:31.893615 30915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.895056 30890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:31.895200 30890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "1861188064d64b27881f03e3cc21d710"
format_stamp: "Formatted at 2026-08-11 22:24:31 on dist-test-slave-4tdj"
server_key: "467dc5dcd1367f19eb0deb87f4f7f657"
server_key_iv: "bea6b41a8713d32dc927b83b0be8eba2"
server_key_version: "encryptionkey@0"
I20260811 22:24:31.895334 30890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:31.905849 30890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:31.906680 30890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:31.906870 30890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:31.907491 30890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:31.908492 30890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:31.908553 30890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.908615 30890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:31.908640 30890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:31.917717 30890 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:42239
I20260811 22:24:31.917745 31138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:42239 every 8 connection(s)
I20260811 22:24:31.918708 30890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:31.926265 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 30890
I20260811 22:24:31.926429 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:24:31.926740 29709 external_mini_cluster.cc:1428] Setting key 6c57eff6fb1c5533c127c1addedddc7d
I20260811 22:24:31.928453 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.920660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:56933 (local address 127.29.3.126:35535)
0811 22:24:31.920812 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.920815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.921914 (+ 1099us) server_negotiation.cc:407] Connection header received
0811 22:24:31.923380 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.923383 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.923448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.923548 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:31.924496 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.925044 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.926194 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.926566 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.926677 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.927929 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:31.927933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:31.927937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:31.927952 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:31.927976 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:31.927986 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:31.928072 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.928276 (+ 204us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.928339 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 22:24:31.929440 31146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.921069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:35535 (local address 127.29.3.67:56933)
0811 22:24:31.921762 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.921805 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:31.923137 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:31.923724 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:31.923732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:31.923823 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:31.924340 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.924353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.925154 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.925157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.926070 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.926079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.926737 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.926740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.926912 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.927625 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:31.927627 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:31.927687 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:31.927781 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:31.928097 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:31.928108 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:31.928115 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:24:31.928252 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 22:24:31.928605 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":105,"threads_started":1}
I20260811 22:24:31.930459 31139 heartbeater.cc:347] Connected to a master server at 127.29.3.126:35535
I20260811 22:24:31.930722 31139 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:31.931262 31139 heartbeater.cc:511] Master 127.29.3.126:35535 requested a full tablet report, sending...
I20260811 22:24:31.932248 30259 ts_manager.cc:194] Registered new tserver with Master: 1861188064d64b27881f03e3cc21d710 (127.29.3.67:42239)
I20260811 22:24:31.932825 30259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.67:56933
I20260811 22:24:31.940766 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:24:31.946866 30334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.942862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.29.3.126:35535)
0811 22:24:31.943034 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.943039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.943199 (+ 160us) server_negotiation.cc:407] Connection header received
0811 22:24:31.943354 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.943357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.943402 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.943487 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:31.943793 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.944328 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.945164 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.945420 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.945517 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.946374 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:31.946377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:31.946379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:31.946392 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:31.946413 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:31.946423 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:31.946542 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.946664 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.946706 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 22:24:31.949908 30259 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:24:31.953227 30259 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:24:31.968956 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.962827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39266 (local address 127.29.3.65:45577)
0811 22:24:31.963474 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.963486 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.963939 (+ 453us) server_negotiation.cc:407] Connection header received
0811 22:24:31.964032 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.964038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.964265 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.964421 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:31.964948 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.966175 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.967700 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.968237 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.968339 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.968544 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.968689 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.968796 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":112,"threads_started":1}
I20260811 22:24:31.968952 31166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.962666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:45577 (local address 127.0.0.1:39266)
0811 22:24:31.963576 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.963640 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:31.963789 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:31.964614 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:31.964620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:31.964646 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:31.964800 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.964809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.966360 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.966363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.967571 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.967582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.968362 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.968365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.968498 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.968529 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:24:31.968645 (+ 116us) client_negotiation.cc:240] Negotiation successful
0811 22:24:31.968785 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":100,"thread_start_us":77,"threads_started":1}
I20260811 22:24:31.969660 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.963026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:42239 (local address 127.0.0.1:44590)
0811 22:24:31.964131 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.964144 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:31.964242 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:31.964844 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:31.964847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:31.964861 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:31.964946 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.964951 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.965904 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.965907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.967344 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.967356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.968228 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.968232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.968341 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.969345 (+ 1004us) client_negotiation.cc:769] Sending connection context
0811 22:24:31.969423 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 22:24:31.969505 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":1041,"thread_start_us":103,"threads_started":1}
I20260811 22:24:31.969738 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.963500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.29.3.67:42239)
0811 22:24:31.964003 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.964014 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.964336 (+ 322us) server_negotiation.cc:407] Connection header received
0811 22:24:31.964418 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.964423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.964585 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.964718 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:31.965050 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.965796 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.967477 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.968111 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.968214 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.968438 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.969475 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.969582 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":149,"threads_started":1}
I20260811 22:24:31.970196 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.962819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39801 (local address 127.0.0.1:60782)
0811 22:24:31.964052 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:31.964065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:31.964170 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:31.964683 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:31.964686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:31.964703 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:31.964806 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.964812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.966321 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.966324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.967590 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:31.967599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.969439 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.969442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:31.969529 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.969552 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:24:31.969616 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 22:24:31.969681 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":103,"thread_start_us":125,"threads_started":1}
I20260811 22:24:31.970317 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:31.963393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.29.3.66:39801)
0811 22:24:31.963833 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:31.963845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:31.964263 (+ 418us) server_negotiation.cc:407] Connection header received
0811 22:24:31.964348 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:31.964353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:31.964532 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:31.964680 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:31.964925 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.966175 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.968443 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:31.968999 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:31.969107 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:31.969298 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:31.970020 (+ 722us) server_negotiation.cc:299] Negotiation successful
0811 22:24:31.970132 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":142,"threads_started":1}
I20260811 22:24:31.972216 30497 tablet_service.cc:1467] Processing CreateTablet for tablet 5284c3ba76d0413291d7d3020729cf35 (DEFAULT_TABLE table=test-table [id=4d8e53c389634620b307bca099af7663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:31.973255 30497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5284c3ba76d0413291d7d3020729cf35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:31.972990 30743 tablet_service.cc:1467] Processing CreateTablet for tablet 5284c3ba76d0413291d7d3020729cf35 (DEFAULT_TABLE table=test-table [id=4d8e53c389634620b307bca099af7663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:31.974153 30743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5284c3ba76d0413291d7d3020729cf35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:31.975569 31012 tablet_service.cc:1467] Processing CreateTablet for tablet 5284c3ba76d0413291d7d3020729cf35 (DEFAULT_TABLE table=test-table [id=4d8e53c389634620b307bca099af7663]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:31.976670 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5284c3ba76d0413291d7d3020729cf35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:31.980331 31173 tablet_bootstrap.cc:492] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: Bootstrap starting.
I20260811 22:24:31.981534 31175 tablet_bootstrap.cc:492] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Bootstrap starting.
I20260811 22:24:31.982904 31173 tablet_bootstrap.cc:654] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:31.983697 31175 tablet_bootstrap.cc:654] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:31.983693 31177 tablet_bootstrap.cc:492] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Bootstrap starting.
I20260811 22:24:31.983794 31173 log.cc:824] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:31.985337 31175 log.cc:824] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:31.986033 31177 tablet_bootstrap.cc:654] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:31.986146 31173 tablet_bootstrap.cc:492] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: No bootstrap required, opened a new log
I20260811 22:24:31.986371 31173 ts_tablet_manager.cc:1397] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260811 22:24:31.986904 31177 log.cc:824] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:31.987488 31175 tablet_bootstrap.cc:492] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: No bootstrap required, opened a new log
I20260811 22:24:31.987673 31175 ts_tablet_manager.cc:1397] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260811 22:24:31.989090 31177 tablet_bootstrap.cc:492] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: No bootstrap required, opened a new log
I20260811 22:24:31.989285 31177 ts_tablet_manager.cc:1397] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:24:31.989923 31173 raft_consensus.cc:348] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:31.990167 31173 raft_consensus.cc:374] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:31.990221 31173 raft_consensus.cc:729] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4853b24c5164eafa000ff5fb96bea25, State: Initialized, Role: FOLLOWER
I20260811 22:24:31.990691 31173 consensus_queue.cc:260] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:31.991379 30629 heartbeater.cc:503] Master 127.29.3.126:35535 was elected leader, sending a full tablet report...
I20260811 22:24:31.991750 31173 ts_tablet_manager.cc:1428] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 22:24:31.992092 31175 raft_consensus.cc:348] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:31.992386 31175 raft_consensus.cc:374] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:31.992445 31175 raft_consensus.cc:729] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e744a4770d1f4cbdacb32b7a846809c5, State: Initialized, Role: FOLLOWER
I20260811 22:24:31.992939 31175 consensus_queue.cc:260] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:31.993640 31177 raft_consensus.cc:348] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:31.993892 30881 heartbeater.cc:503] Master 127.29.3.126:35535 was elected leader, sending a full tablet report...
I20260811 22:24:31.993909 31177 raft_consensus.cc:374] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:31.993959 31177 raft_consensus.cc:729] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1861188064d64b27881f03e3cc21d710, State: Initialized, Role: FOLLOWER
I20260811 22:24:31.994436 31177 consensus_queue.cc:260] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:31.994769 31175 ts_tablet_manager.cc:1428] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260811 22:24:31.995257 31139 heartbeater.cc:503] Master 127.29.3.126:35535 was elected leader, sending a full tablet report...
I20260811 22:24:31.995427 31177 ts_tablet_manager.cc:1428] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260811 22:24:32.032589 30630 tablet.cc:2369] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:24:32.073323 31190 raft_consensus.cc:482] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:32.073589 31190 raft_consensus.cc:504] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:32.074955 31190 leader_election.cc:288] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e744a4770d1f4cbdacb32b7a846809c5 (127.29.3.66:39801), 1861188064d64b27881f03e3cc21d710 (127.29.3.67:42239)
I20260811 22:24:32.079174 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:32.075394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:46691 (local address 127.29.3.67:42239)
0811 22:24:32.075540 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:32.075546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:32.075899 (+ 353us) server_negotiation.cc:407] Connection header received
0811 22:24:32.076024 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:32.076027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:32.076079 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:32.076149 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:32.076521 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.077053 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.078093 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.078596 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.078690 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:32.078763 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:32.078977 (+ 214us) server_negotiation.cc:299] Negotiation successful
0811 22:24:32.079051 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 22:24:32.079171 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:32.075348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:33741 (local address 127.29.3.66:39801)
0811 22:24:32.075484 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:32.075488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:32.075717 (+ 229us) server_negotiation.cc:407] Connection header received
0811 22:24:32.075866 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:32.075869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:32.075933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:32.076002 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:32.076390 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.076919 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.078091 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.078588 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.078688 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:32.078753 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:32.078956 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 22:24:32.079033 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 22:24:32.079197 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:32.075279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:42239 (local address 127.29.3.65:46691)
0811 22:24:32.075778 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:32.075791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:32.075914 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:32.076197 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:32.076200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:32.076242 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:32.076387 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:32.076393 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.077176 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.077178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:32.077971 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:32.077980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.078709 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.078712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:32.078864 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:32.078882 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 22:24:32.078948 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 22:24:32.079023 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":29,"thread_start_us":76,"threads_started":1}
I20260811 22:24:32.079196 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:32.075236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:39801 (local address 127.29.3.65:33741)
0811 22:24:32.075622 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:32.075634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:32.075756 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:32.076050 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:32.076053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:32.076073 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:32.076248 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:32.076255 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.077040 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.077044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:32.077933 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:32.077944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.078707 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.078711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:32.078834 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:32.078865 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 22:24:32.078930 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 22:24:32.079020 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":72,"thread_start_us":137,"threads_started":1}
I20260811 22:24:32.080001 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5284c3ba76d0413291d7d3020729cf35" candidate_uuid: "f4853b24c5164eafa000ff5fb96bea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1861188064d64b27881f03e3cc21d710" is_pre_election: true
I20260811 22:24:32.080008 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5284c3ba76d0413291d7d3020729cf35" candidate_uuid: "f4853b24c5164eafa000ff5fb96bea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e744a4770d1f4cbdacb32b7a846809c5" is_pre_election: true
I20260811 22:24:32.080299 30784 raft_consensus.cc:2454] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4853b24c5164eafa000ff5fb96bea25 in term 0.
I20260811 22:24:32.080297 31051 raft_consensus.cc:2454] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4853b24c5164eafa000ff5fb96bea25 in term 0.
I20260811 22:24:32.080825 30399 leader_election.cc:302] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e744a4770d1f4cbdacb32b7a846809c5, f4853b24c5164eafa000ff5fb96bea25; no voters:
I20260811 22:24:32.081125 31190 raft_consensus.cc:2781] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:32.081218 31190 raft_consensus.cc:482] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:32.081265 31190 raft_consensus.cc:3037] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:32.082378 31190 raft_consensus.cc:504] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:32.082733 31190 leader_election.cc:288] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [CANDIDATE]: Term 1 election: Requested vote from peers e744a4770d1f4cbdacb32b7a846809c5 (127.29.3.66:39801), 1861188064d64b27881f03e3cc21d710 (127.29.3.67:42239)
I20260811 22:24:32.083089 30784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5284c3ba76d0413291d7d3020729cf35" candidate_uuid: "f4853b24c5164eafa000ff5fb96bea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e744a4770d1f4cbdacb32b7a846809c5"
I20260811 22:24:32.083122 31051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5284c3ba76d0413291d7d3020729cf35" candidate_uuid: "f4853b24c5164eafa000ff5fb96bea25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1861188064d64b27881f03e3cc21d710"
I20260811 22:24:32.083217 30784 raft_consensus.cc:3037] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:32.083225 31051 raft_consensus.cc:3037] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:32.084244 31051 raft_consensus.cc:2454] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4853b24c5164eafa000ff5fb96bea25 in term 1.
I20260811 22:24:32.084326 30784 raft_consensus.cc:2454] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4853b24c5164eafa000ff5fb96bea25 in term 1.
I20260811 22:24:32.084596 30404 leader_election.cc:302] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1861188064d64b27881f03e3cc21d710, f4853b24c5164eafa000ff5fb96bea25; no voters:
I20260811 22:24:32.084780 31190 raft_consensus.cc:2781] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:32.085026 31190 raft_consensus.cc:686] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [term 1 LEADER]: Becoming Leader. State: Replica: f4853b24c5164eafa000ff5fb96bea25, State: Running, Role: LEADER
I20260811 22:24:32.085412 31190 consensus_queue.cc:237] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } }
I20260811 22:24:32.088778 30259 catalog_manager.cc:5495] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4853b24c5164eafa000ff5fb96bea25 (127.29.3.65). New cstate: current_term: 1 leader_uuid: "f4853b24c5164eafa000ff5fb96bea25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4853b24c5164eafa000ff5fb96bea25" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 45577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20260811 22:24:32.125667 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:32.121879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39280 (local address 127.29.3.65:45577)
0811 22:24:32.122034 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:32.122039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:32.122055 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:24:32.122172 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:32.122175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:32.122253 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:32.122389 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:24:32.122648 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.123170 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.124052 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:32.124295 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:32.124387 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:32.124752 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:24:32.125191 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:24:32.125344 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:32.125477 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 22:24:32.125528 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 22:24:32.132079 31051 raft_consensus.cc:1261] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 1 FOLLOWER]: Refusing update from remote peer f4853b24c5164eafa000ff5fb96bea25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:32.132097 30784 raft_consensus.cc:1261] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 1 FOLLOWER]: Refusing update from remote peer f4853b24c5164eafa000ff5fb96bea25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:32.132839 31196 consensus_queue.cc:1035] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e744a4770d1f4cbdacb32b7a846809c5" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 39801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:32.133028 31190 consensus_queue.cc:1035] T 5284c3ba76d0413291d7d3020729cf35 P f4853b24c5164eafa000ff5fb96bea25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1861188064d64b27881f03e3cc21d710" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:32.141691 31204 mvcc.cc:205] Tried to move back new op lower bound from 7317451047445311488 to 7317451047264772096. Current Snapshot: MvccSnapshot[applied={T|T < 7317451047445311488}]
I20260811 22:24:32.141899 31202 mvcc.cc:205] Tried to move back new op lower bound from 7317451047445311488 to 7317451047264772096. Current Snapshot: MvccSnapshot[applied={T|T < 7317451047445311488}]
I20260811 22:24:32.149085 30248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:32.149266 30248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:32.152238 30248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 289054f760e54f3a84b4581b40592760: Sending DeleteTablet for 3 replicas of tablet 5284c3ba76d0413291d7d3020729cf35
I20260811 22:24:32.153321 30497 tablet_service.cc:1514] Processing DeleteTablet for tablet 5284c3ba76d0413291d7d3020729cf35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:32 UTC) from {username='slave'} at 127.0.0.1:39266
I20260811 22:24:32.153337 31012 tablet_service.cc:1514] Processing DeleteTablet for tablet 5284c3ba76d0413291d7d3020729cf35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:32 UTC) from {username='slave'} at 127.0.0.1:44590
I20260811 22:24:32.153344 30743 tablet_service.cc:1514] Processing DeleteTablet for tablet 5284c3ba76d0413291d7d3020729cf35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:32 UTC) from {username='slave'} at 127.0.0.1:60782
I20260811 22:24:32.153935 31215 tablet_replica.cc:330] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: stopping tablet replica
I20260811 22:24:32.154178 31213 tablet_replica.cc:330] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: stopping tablet replica
I20260811 22:24:32.154520 31215 raft_consensus.cc:2229] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:32.154559 31213 raft_consensus.cc:2229] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:32.154971 31213 raft_consensus.cc:2258] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:32.154973 31215 raft_consensus.cc:2258] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:32.155129 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30372
I20260811 22:24:32.159986 31213 ts_tablet_manager.cc:1905] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:32.160447 31215 ts_tablet_manager.cc:1905] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:32.162580 31213 ts_tablet_manager.cc:1918] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:32.162688 31213 log.cc:1196] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/5284c3ba76d0413291d7d3020729cf35
I20260811 22:24:32.162953 31213 ts_tablet_manager.cc:1939] T 5284c3ba76d0413291d7d3020729cf35 P e744a4770d1f4cbdacb32b7a846809c5: Deleting consensus metadata
I20260811 22:24:32.163223 31215 ts_tablet_manager.cc:1918] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:32.163323 31215 log.cc:1196] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/5284c3ba76d0413291d7d3020729cf35
I20260811 22:24:32.163592 31215 ts_tablet_manager.cc:1939] T 5284c3ba76d0413291d7d3020729cf35 P 1861188064d64b27881f03e3cc21d710: Deleting consensus metadata
I20260811 22:24:32.164291 30228 catalog_manager.cc:4841] TS e744a4770d1f4cbdacb32b7a846809c5 (127.29.3.66:39801): tablet 5284c3ba76d0413291d7d3020729cf35 (table test-table [id=4d8e53c389634620b307bca099af7663]) successfully deleted
I20260811 22:24:32.164729 30229 catalog_manager.cc:4841] TS 1861188064d64b27881f03e3cc21d710 (127.29.3.67:42239): tablet 5284c3ba76d0413291d7d3020729cf35 (table test-table [id=4d8e53c389634620b307bca099af7663]) successfully deleted
W20260811 22:24:32.167168 30234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:45577 (error 108)
I20260811 22:24:32.168009 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30634
I20260811 22:24:32.168260 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:32.167970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:45577 (local address 127.0.0.1:39282)
0811 22:24:32.168108 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:32.168179 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260811 22:24:32.205426 30234 catalog_manager.cc:4584] TS f4853b24c5164eafa000ff5fb96bea25 (127.29.3.65:45577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5284c3ba76d0413291d7d3020729cf35: Network error: Client connection negotiation failed: client connection to 127.29.3.65:45577: connect: Connection refused (error 111)
I20260811 22:24:32.212330 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30890
I20260811 22:24:32.232513 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 30199
2026-08-11T22:24:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1176 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:32 dist-test-slave-4tdj krb5kdc[31225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:32 dist-test-slave-4tdj krb5kdc[31225](info): set up 2 sockets
Aug 11 22:24:32 dist-test-slave-4tdj krb5kdc[31225](info): commencing operation
krb5kdc: starting...
W20260811 22:24:33.432320 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:33 dist-test-slave-4tdj krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:24:33.565567 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36363
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:36363
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:33.683800 31248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:33.684108 31248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:33.684177 31248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:33.688449 31248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:33.688540 31248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:33.688567 31248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:33.688589 31248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:33.688619 31248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:33.694345 31248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:36363
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:36363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:33.695523 31248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:33.696594 31248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:33.705515 31260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:33.705595 31264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:33.705766 31248 server_base.cc:1034] running on GCE node
W20260811 22:24:33.705616 31261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:33.706580 31248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:33.707777 31248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:33.708961 31248 hybrid_clock.cc:648] HybridClock initialized: now 1786487073708932 us; error 44 us; skew 500 ppm
Aug 11 22:24:33 dist-test-slave-4tdj krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487073, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:33.711920 31248 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:33.712909 31248 webserver.cc:460] Webserver started at http://127.29.3.126:39687/ using document root <none> and password file <none>
I20260811 22:24:33.713505 31248 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:33.713663 31248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:33.713932 31248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:33.716234 31248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "383cbdffda0b4dec983f8dbbd78ab38a"
format_stamp: "Formatted at 2026-08-11 22:24:33 on dist-test-slave-4tdj"
server_key: "110a730401cd4bda4996da627f5dddf2"
server_key_iv: "5fa6810a801cc6eea5ae23f496d4f18d"
server_key_version: "encryptionkey@0"
I20260811 22:24:33.716843 31248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "383cbdffda0b4dec983f8dbbd78ab38a"
format_stamp: "Formatted at 2026-08-11 22:24:33 on dist-test-slave-4tdj"
server_key: "110a730401cd4bda4996da627f5dddf2"
server_key_iv: "5fa6810a801cc6eea5ae23f496d4f18d"
server_key_version: "encryptionkey@0"
I20260811 22:24:33.721496 31248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:24:33.724419 31281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:33.725865 31248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:33.726006 31248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "383cbdffda0b4dec983f8dbbd78ab38a"
format_stamp: "Formatted at 2026-08-11 22:24:33 on dist-test-slave-4tdj"
server_key: "110a730401cd4bda4996da627f5dddf2"
server_key_iv: "5fa6810a801cc6eea5ae23f496d4f18d"
server_key_version: "encryptionkey@0"
I20260811 22:24:33.726127 31248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:33.742828 31248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:33.746061 31248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:33.746248 31248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:33.754531 31248 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:36363
I20260811 22:24:33.754562 31382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:36363 every 8 connection(s)
I20260811 22:24:33.755612 31248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:33.758610 31383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:33.763804 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31248
I20260811 22:24:33.763969 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:33.764310 29709 external_mini_cluster.cc:1428] Setting key 3b20592e2be761f063bcf0485577f7d8
I20260811 22:24:33.765087 31383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: Bootstrap starting.
I20260811 22:24:33.768922 31383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:33.769729 31383 log.cc:824] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:33.772105 31383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: No bootstrap required, opened a new log
Aug 11 22:24:33 dist-test-slave-4tdj krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:33.776592 31383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36363 } }
I20260811 22:24:33.776904 31383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:33.776968 31383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 383cbdffda0b4dec983f8dbbd78ab38a, State: Initialized, Role: FOLLOWER
I20260811 22:24:33.777465 31383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36363 } }
I20260811 22:24:33.777614 31383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:33.777684 31383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:33.777804 31383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:33.778937 31383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36363 } }
I20260811 22:24:33.779184 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:33.765895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.29.3.126:36363)
0811 22:24:33.766392 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:33.766410 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:33.766449 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:24:33.767150 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:33.767169 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:33.767529 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:33.767852 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:33.768775 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:33.769845 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:33.770942 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:33.771419 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:33.771536 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:33.774345 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:33.774351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:33.774367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:33.774397 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:33.776081 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:33.776579 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:33.776584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:33.776589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:33.776738 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:33.777058 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:33.777062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:33.777064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:33.777504 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:33.777742 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:33.778001 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 22:24:33.778347 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":160,"threads_started":1}
I20260811 22:24:33.779304 31383 leader_election.cc:302] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 383cbdffda0b4dec983f8dbbd78ab38a; no voters:
I20260811 22:24:33.779652 31383 leader_election.cc:288] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:33.779765 31394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:33.779944 31394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [term 1 LEADER]: Becoming Leader. State: Replica: 383cbdffda0b4dec983f8dbbd78ab38a, State: Running, Role: LEADER
I20260811 22:24:33.781073 31383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:33.781257 31394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36363 } }
I20260811 22:24:33.783118 31396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 383cbdffda0b4dec983f8dbbd78ab38a. Latest consensus state: current_term: 1 leader_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36363 } } }
I20260811 22:24:33.783118 31398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "383cbdffda0b4dec983f8dbbd78ab38a" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 36363 } } }
I20260811 22:24:33.783286 31396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:33.783286 31398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a [sys.catalog]: This master's current role is: LEADER
W20260811 22:24:33.786052 31420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:24:33.786154 31420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:24:33.786270 31421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:33.787236 31421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:33.793366 31421 catalog_manager.cc:1347] Generated new cluster ID: 819bba5d0db241bead817ab23e8d6f4e
I20260811 22:24:33.793467 31421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:33.805630 31421 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:33.807125 31421 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:33.815541 31421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: Generated new TSK 0
I20260811 22:24:33.816489 31421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.3.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
Entry for principal kudu/127.29.3.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab.
I20260811 22:24:33.857141 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36363
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:33.981424 31430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:33.981798 31430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:33.981853 31430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:33.986363 31430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:33.986445 31430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:33.986507 31430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:33.992354 31430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.65.keytab
--principal=kudu/127.29.3.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:33.993541 31430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:33.994709 31430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:34.005203 31437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.005205 31440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.005205 31438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:34.005657 31430 server_base.cc:1034] running on GCE node
I20260811 22:24:34.006132 31430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:34.006910 31430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:34.008078 31430 hybrid_clock.cc:648] HybridClock initialized: now 1786487074008055 us; error 37 us; skew 500 ppm
Aug 11 22:24:34 dist-test-slave-4tdj krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:34.011103 31430 init.cc:377] Logged in from keytab as kudu/127.29.3.65@KRBTEST.COM (short username kudu)
I20260811 22:24:34.012133 31430 webserver.cc:460] Webserver started at http://127.29.3.65:41013/ using document root <none> and password file <none>
I20260811 22:24:34.012737 31430 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:34.012805 31430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:34.013023 31430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:34.015379 31430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "2e7519c751e1467dbdeca862e109b5bc"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "f24238a31ad12137637f58c07f00a759"
server_key_iv: "bc3650e585993dbe5c38a81d54a220b5"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.015978 31430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "2e7519c751e1467dbdeca862e109b5bc"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "f24238a31ad12137637f58c07f00a759"
server_key_iv: "bc3650e585993dbe5c38a81d54a220b5"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.020211 31430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:34.022959 31453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.024308 31430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:34.024428 31430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "2e7519c751e1467dbdeca862e109b5bc"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "f24238a31ad12137637f58c07f00a759"
server_key_iv: "bc3650e585993dbe5c38a81d54a220b5"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.024556 31430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:34.037617 31430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:34.041131 31430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:34.041347 31430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:34.042006 31430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:34.042984 31430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:34.043036 31430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.043079 31430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:34.043093 31430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.052640 31430 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:41723
I20260811 22:24:34.052728 31684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:41723 every 8 connection(s)
I20260811 22:24:34.053602 31430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:34.055205 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31430
I20260811 22:24:34.055334 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:24:34.055681 29709 external_mini_cluster.cc:1428] Setting key d868128930fb0b1d495572ea552a8d73
Aug 11 22:24:34 dist-test-slave-4tdj krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.65@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.66@KRBTEST.COM; defaulting to no policy
I20260811 22:24:34.068876 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.055509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:36637 (local address 127.29.3.126:36363)
0811 22:24:34.055683 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.055688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.056924 (+ 1236us) server_negotiation.cc:407] Connection header received
0811 22:24:34.058088 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.058090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.058137 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.058209 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:34.059863 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.060407 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.061598 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.061961 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.062078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.064854 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:34.064858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:34.064862 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:34.064899 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:34.066593 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.067162 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.067167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.067169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.067275 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.067612 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.067615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.067617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.067895 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:34.068008 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.068563 (+ 555us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.068729 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:24:34.069773 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.055949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36363 (local address 127.29.3.65:36637)
0811 22:24:34.056773 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.056821 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.057830 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.058458 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.058467 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.058969 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:34.059705 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.059727 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.060543 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.060547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.061468 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.061478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.062150 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.062153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.062386 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.063012 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:34.063031 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:34.064633 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:34.066736 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:34.066741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:34.066758 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:34.067050 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:34.067367 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:34.067369 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:34.067370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:34.067523 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:34.068006 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:34.068013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:34.068316 (+ 303us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.068488 (+ 172us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.068791 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.66@KRBTEST.COM" created.
I20260811 22:24:34.071264 31685 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36363
I20260811 22:24:34.071547 31685 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:34.072131 31685 heartbeater.cc:511] Master 127.29.3.126:36363 requested a full tablet report, sending...
I20260811 22:24:34.074028 31310 ts_manager.cc:194] Registered new tserver with Master: 2e7519c751e1467dbdeca862e109b5bc (127.29.3.65:41723)
I20260811 22:24:34.075650 31310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.65@KRBTEST.COM'} at 127.29.3.65:36637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
Entry for principal kudu/127.29.3.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab.
I20260811 22:24:34.090134 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36363
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:34.209651 31697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:34.209955 31697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:34.210009 31697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:34.214206 31697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:34.214285 31697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:34.214408 31697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:34.220023 31697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.66.keytab
--principal=kudu/127.29.3.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:34.221132 31697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:34.222162 31697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:34.232188 31712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.232193 31707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.232201 31708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:34.232391 31697 server_base.cc:1034] running on GCE node
I20260811 22:24:34.232973 31697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:34.233655 31697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:34.234848 31697 hybrid_clock.cc:648] HybridClock initialized: now 1786487074234830 us; error 35 us; skew 500 ppm
Aug 11 22:24:34 dist-test-slave-4tdj krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:34.237811 31697 init.cc:377] Logged in from keytab as kudu/127.29.3.66@KRBTEST.COM (short username kudu)
I20260811 22:24:34.238799 31697 webserver.cc:460] Webserver started at http://127.29.3.66:46123/ using document root <none> and password file <none>
I20260811 22:24:34.239358 31697 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:34.239418 31697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:34.239590 31697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:34.241717 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "fb5d9e8ac32b469facefef4e9bde8e2f"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "17f3c2c95d0a2375803bfd7c19b16c91"
server_key_iv: "4a1e4b17e4da3878cd176ad8a37ce765"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.242323 31697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "fb5d9e8ac32b469facefef4e9bde8e2f"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "17f3c2c95d0a2375803bfd7c19b16c91"
server_key_iv: "4a1e4b17e4da3878cd176ad8a37ce765"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.246243 31697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:24:34.248857 31729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.250236 31697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:34.250396 31697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "fb5d9e8ac32b469facefef4e9bde8e2f"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "17f3c2c95d0a2375803bfd7c19b16c91"
server_key_iv: "4a1e4b17e4da3878cd176ad8a37ce765"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.250520 31697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:34.270254 31697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:34.273559 31697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:34.273741 31697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:34.274390 31697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:34.275491 31697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:34.275544 31697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.275590 31697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:34.275611 31697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.284580 31697 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:45011
I20260811 22:24:34.284628 31928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:45011 every 8 connection(s)
I20260811 22:24:34.285529 31697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:34.288239 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31697
I20260811 22:24:34.288367 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:24:34.288703 29709 external_mini_cluster.cc:1428] Setting key 3dd9e8e37720095faa11d756339b46bb
Aug 11 22:24:34 dist-test-slave-4tdj krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.66@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.3.67@KRBTEST.COM; defaulting to no policy
I20260811 22:24:34.300797 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.287429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:60987 (local address 127.29.3.126:36363)
0811 22:24:34.287571 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.287575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.288791 (+ 1216us) server_negotiation.cc:407] Connection header received
0811 22:24:34.289909 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.289912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.289960 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.290049 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:34.291407 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.291948 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.293641 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.293891 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.293981 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.297143 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:34.297147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:34.297149 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:34.297177 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:34.298580 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.299199 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.299202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.299204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.299289 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.299671 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.299674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.299675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.299892 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:34.299973 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.300539 (+ 566us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.300662 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:24:34.301703 31933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.287854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36363 (local address 127.29.3.66:60987)
0811 22:24:34.288634 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.288681 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.289659 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.290285 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.290318 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.290720 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:34.291251 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.291267 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.292083 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.292086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.293497 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.293511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.294068 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.294072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.294324 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.294966 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:34.294987 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:34.296923 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:34.298705 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:34.298713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:34.298731 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:34.299093 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:34.299379 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:34.299382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:34.299384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:34.299572 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:34.299996 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:34.300004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:34.300302 (+ 298us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.300478 (+ 176us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.300821 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.67@KRBTEST.COM" created.
I20260811 22:24:34.303067 31930 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36363
I20260811 22:24:34.303345 31930 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:34.303882 31930 heartbeater.cc:511] Master 127.29.3.126:36363 requested a full tablet report, sending...
I20260811 22:24:34.305133 31310 ts_manager.cc:194] Registered new tserver with Master: fb5d9e8ac32b469facefef4e9bde8e2f (127.29.3.66:45011)
I20260811 22:24:34.305768 31310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.66@KRBTEST.COM'} at 127.29.3.66:60987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
Entry for principal kudu/127.29.3.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab.
I20260811 22:24:34.322595 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36363
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:34.441720 31937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:34.442021 31937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:34.442066 31937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:34.446214 31937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:34.446331 31937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:34.446403 31937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:34.452001 31937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.67.keytab
--principal=kudu/127.29.3.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:36363
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:34.453169 31937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:34.454321 31937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:34.464504 31946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.464537 31945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:34.464632 31937 server_base.cc:1034] running on GCE node
W20260811 22:24:34.464537 31950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:34.465161 31937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:34.465883 31937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:34.467059 31937 hybrid_clock.cc:648] HybridClock initialized: now 1786487074467021 us; error 50 us; skew 500 ppm
Aug 11 22:24:34 dist-test-slave-4tdj krb5kdc[31225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:34.470067 31937 init.cc:377] Logged in from keytab as kudu/127.29.3.67@KRBTEST.COM (short username kudu)
I20260811 22:24:34.471096 31937 webserver.cc:460] Webserver started at http://127.29.3.67:33375/ using document root <none> and password file <none>
I20260811 22:24:34.471681 31937 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:34.471747 31937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:34.471930 31937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:34.474136 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "604e29a52ad843aa88a9016471cff96a"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "e74f529ac45b301c01ac913648f4a02d"
server_key_iv: "edda654d7cb71e8ae865a2253d2735c2"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.474761 31937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "604e29a52ad843aa88a9016471cff96a"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "e74f529ac45b301c01ac913648f4a02d"
server_key_iv: "edda654d7cb71e8ae865a2253d2735c2"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.478861 31937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:24:34.481513 31962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.483263 31937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:34.483383 31937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "604e29a52ad843aa88a9016471cff96a"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "e74f529ac45b301c01ac913648f4a02d"
server_key_iv: "edda654d7cb71e8ae865a2253d2735c2"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.483512 31937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:34.506150 31937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:34.509616 31937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:34.509793 31937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:34.510429 31937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:34.511323 31937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:34.511384 31937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.511425 31937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:34.511445 31937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.521963 31937 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:36853
I20260811 22:24:34.521991 32168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:36853 every 8 connection(s)
I20260811 22:24:34.523080 31937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:34.530732 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 31937
I20260811 22:24:34.530885 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:24:34.531173 29709 external_mini_cluster.cc:1428] Setting key cd6578b0ee711a362b86bb1c62de8a07
Aug 11 22:24:34 dist-test-slave-4tdj krb5kdc[31225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487074, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.67@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:34.537093 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.525044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:50971 (local address 127.29.3.126:36363)
0811 22:24:34.525179 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.525183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.526243 (+ 1060us) server_negotiation.cc:407] Connection header received
0811 22:24:34.527157 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.527159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.527215 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.527290 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:34.528647 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.529240 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.530521 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.530767 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.530896 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.533636 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:34.533638 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:34.533640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:34.533667 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:34.535080 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.535578 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.535582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.535584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.535665 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.535988 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.535991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.535993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.536191 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:34.536278 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.536797 (+ 519us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.536927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 22:24:34.537961 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.525404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:36363 (local address 127.29.3.67:50971)
0811 22:24:34.526074 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.526120 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.526965 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.527524 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.527532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.527945 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:34.528478 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.528492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.529360 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.529363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.530404 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.530415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.530943 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.530947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.531145 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.531752 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:24:34.531771 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:34.533454 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:34.535191 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:34.535197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:34.535217 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:34.535475 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:34.535749 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:24:34.535752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:24:34.535754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:24:34.535901 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:24:34.536291 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:34.536298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:34.536562 (+ 264us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.536741 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.537112 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":109,"threads_started":1}
I20260811 22:24:34.539215 32169 heartbeater.cc:347] Connected to a master server at 127.29.3.126:36363
I20260811 22:24:34.539479 32169 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:34.540045 32169 heartbeater.cc:511] Master 127.29.3.126:36363 requested a full tablet report, sending...
I20260811 22:24:34.541106 31310 ts_manager.cc:194] Registered new tserver with Master: 604e29a52ad843aa88a9016471cff96a (127.29.3.67:36853)
I20260811 22:24:34.541676 31310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.3.67@KRBTEST.COM'} at 127.29.3.67:50971
I20260811 22:24:34.545028 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:24:34.554111 31387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.547007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48526 (local address 127.29.3.126:36363)
0811 22:24:34.547160 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.547165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.547287 (+ 122us) server_negotiation.cc:407] Connection header received
0811 22:24:34.547392 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.547395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.547436 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.547501 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:34.548148 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.548657 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.549456 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.549674 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.549755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.551253 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:34.551257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:34.551260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:34.551288 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:34.552485 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.552884 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.552886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.552887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.552957 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:34.553142 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:34.553144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:34.553145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:34.553383 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:34.553488 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.553785 (+ 297us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.553928 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 22:24:34.557286 31310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48526:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:24:34.560508 31310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:24:34.575744 32193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.569527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:41723 (local address 127.0.0.1:54076)
0811 22:24:34.570409 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.570424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.570528 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.571582 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.571585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.571611 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:34.571771 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.571779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.573179 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.573183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.574329 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.574339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.575152 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.575156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.575274 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.575313 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.575431 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.575576 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":48,"spinlock_wait_cycles":2688,"thread_start_us":104,"threads_started":1}
I20260811 22:24:34.575749 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.570570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.29.3.65:41723)
0811 22:24:34.570976 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.570981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.570999 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:24:34.571076 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.571081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.571279 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.571426 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:34.571904 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.573055 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.574449 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.575029 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.575132 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.575318 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.575475 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.575577 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":141,"threads_started":1}
I20260811 22:24:34.576116 32191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.569419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:36853 (local address 127.0.0.1:44470)
0811 22:24:34.570103 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.570169 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.570331 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.571415 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.571422 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.571446 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:34.571576 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.571582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.573074 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.573078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.574232 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.574242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.575559 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.575562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.575766 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.575817 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.575888 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.575976 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":69,"thread_start_us":95,"threads_started":1}
I20260811 22:24:34.576342 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.569753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45011 (local address 127.0.0.1:45844)
0811 22:24:34.570442 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.570455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.570531 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.571582 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.571585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.571611 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:34.571774 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.571779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.572799 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.572803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.574193 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.574213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.575556 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.575560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.575789 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.575964 (+ 175us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.576031 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.576111 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":163,"spinlock_wait_cycles":6272,"thread_start_us":108,"threads_started":1}
I20260811 22:24:34.576581 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.569546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.29.3.67:36853)
0811 22:24:34.570797 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.570810 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.570834 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:24:34.570914 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.570919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.571151 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.571311 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:34.571695 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.572874 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.574402 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.575238 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.575346 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.575488 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.576317 (+ 829us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.576427 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":177,"threads_started":1}
I20260811 22:24:34.576887 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.570489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.29.3.66:45011)
0811 22:24:34.570963 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.570975 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.570999 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:24:34.571076 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.571081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.571279 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.571426 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:34.571926 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.572659 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.574354 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.575238 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.575346 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.575488 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.576620 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.576743 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":124,"threads_started":1}
I20260811 22:24:34.579180 31554 tablet_service.cc:1467] Processing CreateTablet for tablet 9c556305b0a249b2895e064410959917 (DEFAULT_TABLE table=test-table [id=592c2f2bf2f549e38b18a235229ee9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:34.579175 32057 tablet_service.cc:1467] Processing CreateTablet for tablet 9c556305b0a249b2895e064410959917 (DEFAULT_TABLE table=test-table [id=592c2f2bf2f549e38b18a235229ee9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:34.580188 31554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c556305b0a249b2895e064410959917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:34.580196 32057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c556305b0a249b2895e064410959917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:34.583560 31816 tablet_service.cc:1467] Processing CreateTablet for tablet 9c556305b0a249b2895e064410959917 (DEFAULT_TABLE table=test-table [id=592c2f2bf2f549e38b18a235229ee9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:34.584565 31816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c556305b0a249b2895e064410959917. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:34.588174 32206 tablet_bootstrap.cc:492] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Bootstrap starting.
I20260811 22:24:34.589781 32210 tablet_bootstrap.cc:492] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc: Bootstrap starting.
I20260811 22:24:34.591449 32206 tablet_bootstrap.cc:654] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:34.591835 32210 tablet_bootstrap.cc:654] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:34.592469 32206 log.cc:824] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:34.592715 32210 log.cc:824] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:34.592980 32211 tablet_bootstrap.cc:492] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Bootstrap starting.
I20260811 22:24:34.594660 32206 tablet_bootstrap.cc:492] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: No bootstrap required, opened a new log
I20260811 22:24:34.594877 32206 ts_tablet_manager.cc:1397] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260811 22:24:34.594939 32210 tablet_bootstrap.cc:492] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc: No bootstrap required, opened a new log
I20260811 22:24:34.595147 32210 ts_tablet_manager.cc:1397] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:24:34.595841 32211 tablet_bootstrap.cc:654] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:34.596766 32211 log.cc:824] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:34.598161 32206 raft_consensus.cc:348] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.598462 32206 raft_consensus.cc:374] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:34.598521 32206 raft_consensus.cc:729] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604e29a52ad843aa88a9016471cff96a, State: Initialized, Role: FOLLOWER
I20260811 22:24:34.598978 32206 consensus_queue.cc:260] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.599162 32211 tablet_bootstrap.cc:492] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: No bootstrap required, opened a new log
I20260811 22:24:34.599361 32211 ts_tablet_manager.cc:1397] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 22:24:34.599649 32210 raft_consensus.cc:348] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.599923 32210 raft_consensus.cc:374] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:34.599973 32210 raft_consensus.cc:729] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e7519c751e1467dbdeca862e109b5bc, State: Initialized, Role: FOLLOWER
I20260811 22:24:34.600471 32210 consensus_queue.cc:260] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.600718 32206 ts_tablet_manager.cc:1428] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:24:34.601079 32169 heartbeater.cc:503] Master 127.29.3.126:36363 was elected leader, sending a full tablet report...
I20260811 22:24:34.601483 32210 ts_tablet_manager.cc:1428] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260811 22:24:34.601832 31685 heartbeater.cc:503] Master 127.29.3.126:36363 was elected leader, sending a full tablet report...
I20260811 22:24:34.604979 32211 raft_consensus.cc:348] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.605345 32211 raft_consensus.cc:374] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:34.605424 32211 raft_consensus.cc:729] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb5d9e8ac32b469facefef4e9bde8e2f, State: Initialized, Role: FOLLOWER
I20260811 22:24:34.605919 32211 consensus_queue.cc:260] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.606750 31930 heartbeater.cc:503] Master 127.29.3.126:36363 was elected leader, sending a full tablet report...
I20260811 22:24:34.607071 32211 ts_tablet_manager.cc:1428] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260811 22:24:34.656857 32216 raft_consensus.cc:482] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:34.657097 32216 raft_consensus.cc:504] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.658397 32216 leader_election.cc:288] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e7519c751e1467dbdeca862e109b5bc (127.29.3.65:41723), fb5d9e8ac32b469facefef4e9bde8e2f (127.29.3.66:45011)
I20260811 22:24:34.662519 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.658776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:57709 (local address 127.29.3.66:45011)
0811 22:24:34.658909 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.658914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.658932 (+ 18us) server_negotiation.cc:407] Connection header received
0811 22:24:34.659129 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.659133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.659195 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.659285 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:34.659618 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.660234 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.661390 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.661931 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.662007 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.662065 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.662323 (+ 258us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.662388 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:24:34.662525 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.658681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:45011 (local address 127.29.3.67:57709)
0811 22:24:34.658843 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.658861 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.658992 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.659329 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.659332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.659358 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:34.659503 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.659510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.660357 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.660361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.661270 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.661279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.662054 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.662057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.662166 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.662194 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.662285 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.662380 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"spinlock_wait_cycles":1920}
I20260811 22:24:34.662915 32202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.658801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:43997 (local address 127.29.3.65:41723)
0811 22:24:34.658936 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.658941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.659383 (+ 442us) server_negotiation.cc:407] Connection header received
0811 22:24:34.659469 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.659472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.659566 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.659641 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:34.659959 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.660602 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.661768 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.662348 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.662437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.662529 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.662719 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.662802 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:24:34.663239 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.658689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:41723 (local address 127.29.3.67:43997)
0811 22:24:34.659275 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.659288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:34.659385 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:34.659683 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:34.659685 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:34.659704 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:34.659820 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.659826 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.660737 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.660740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.661647 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:34.661658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.662468 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.662472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:34.662565 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.662589 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 22:24:34.663061 (+ 472us) client_negotiation.cc:240] Negotiation successful
0811 22:24:34.663139 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":175,"thread_start_us":158,"threads_started":1}
I20260811 22:24:34.663264 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c556305b0a249b2895e064410959917" candidate_uuid: "604e29a52ad843aa88a9016471cff96a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" is_pre_election: true
I20260811 22:24:34.663609 31859 raft_consensus.cc:2454] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604e29a52ad843aa88a9016471cff96a in term 0.
I20260811 22:24:34.663864 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c556305b0a249b2895e064410959917" candidate_uuid: "604e29a52ad843aa88a9016471cff96a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7519c751e1467dbdeca862e109b5bc" is_pre_election: true
I20260811 22:24:34.664170 31596 raft_consensus.cc:2454] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 604e29a52ad843aa88a9016471cff96a in term 0.
I20260811 22:24:34.664156 31966 leader_election.cc:302] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604e29a52ad843aa88a9016471cff96a, fb5d9e8ac32b469facefef4e9bde8e2f; no voters:
I20260811 22:24:34.664448 32216 raft_consensus.cc:2781] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:34.664593 32216 raft_consensus.cc:482] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:34.664660 32216 raft_consensus.cc:3037] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:34.665808 32216 raft_consensus.cc:504] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.666174 32216 leader_election.cc:288] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [CANDIDATE]: Term 1 election: Requested vote from peers 2e7519c751e1467dbdeca862e109b5bc (127.29.3.65:41723), fb5d9e8ac32b469facefef4e9bde8e2f (127.29.3.66:45011)
I20260811 22:24:34.666584 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c556305b0a249b2895e064410959917" candidate_uuid: "604e29a52ad843aa88a9016471cff96a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e7519c751e1467dbdeca862e109b5bc"
I20260811 22:24:34.666611 31859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c556305b0a249b2895e064410959917" candidate_uuid: "604e29a52ad843aa88a9016471cff96a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f"
I20260811 22:24:34.666724 31596 raft_consensus.cc:3037] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:34.666749 31859 raft_consensus.cc:3037] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:34.667930 31859 raft_consensus.cc:2454] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604e29a52ad843aa88a9016471cff96a in term 1.
I20260811 22:24:34.668154 31596 raft_consensus.cc:2454] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 604e29a52ad843aa88a9016471cff96a in term 1.
I20260811 22:24:34.668216 31966 leader_election.cc:302] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604e29a52ad843aa88a9016471cff96a, fb5d9e8ac32b469facefef4e9bde8e2f; no voters:
I20260811 22:24:34.668396 32216 raft_consensus.cc:2781] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:34.668658 32216 raft_consensus.cc:686] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 1 LEADER]: Becoming Leader. State: Replica: 604e29a52ad843aa88a9016471cff96a, State: Running, Role: LEADER
I20260811 22:24:34.669037 32216 consensus_queue.cc:237] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } }
I20260811 22:24:34.672500 31309 catalog_manager.cc:5495] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a reported cstate change: term changed from 0 to 1, leader changed from <none> to 604e29a52ad843aa88a9016471cff96a (127.29.3.67). New cstate: current_term: 1 leader_uuid: "604e29a52ad843aa88a9016471cff96a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "604e29a52ad843aa88a9016471cff96a" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 } health_report { overall_health: UNKNOWN } } }
I20260811 22:24:34.700557 32196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.696877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44472 (local address 127.29.3.67:36853)
0811 22:24:34.697038 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:34.697042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:34.697055 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:24:34.697140 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:34.697142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:34.697219 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:34.697314 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:24:34.697577 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.698132 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.699076 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:34.699304 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:34.699384 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:34.699727 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:24:34.700165 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:24:34.700272 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:34.700365 (+ 93us) server_negotiation.cc:299] Negotiation successful
0811 22:24:34.700417 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 22:24:34.706676 31859 raft_consensus.cc:1261] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 1 FOLLOWER]: Refusing update from remote peer 604e29a52ad843aa88a9016471cff96a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:34.706676 31596 raft_consensus.cc:1261] T 9c556305b0a249b2895e064410959917 P 2e7519c751e1467dbdeca862e109b5bc [term 1 FOLLOWER]: Refusing update from remote peer 604e29a52ad843aa88a9016471cff96a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:34.707412 32216 consensus_queue.cc:1035] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb5d9e8ac32b469facefef4e9bde8e2f" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 45011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:34.707659 32226 consensus_queue.cc:1035] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e7519c751e1467dbdeca862e109b5bc" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:34.715195 32230 mvcc.cc:205] Tried to move back new op lower bound from 7317451057990713344 to 7317451057847267328. Current Snapshot: MvccSnapshot[applied={T|T < 7317451057990713344}]
I20260811 22:24:34.725937 31309 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:34.726115 31309 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48526:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:34.728927 31309 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 383cbdffda0b4dec983f8dbbd78ab38a: Sending DeleteTablet for 3 replicas of tablet 9c556305b0a249b2895e064410959917
I20260811 22:24:34.729769 32057 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c556305b0a249b2895e064410959917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:34 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:44470
I20260811 22:24:34.730003 31816 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c556305b0a249b2895e064410959917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:34 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:45844
I20260811 22:24:34.730382 31554 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c556305b0a249b2895e064410959917 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:34 UTC) from {username='kudu', principal='kudu/127.29.3.126@KRBTEST.COM'} at 127.0.0.1:54076
I20260811 22:24:34.730605 32250 tablet_replica.cc:330] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: stopping tablet replica
I20260811 22:24:34.730791 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31430
I20260811 22:24:34.746682 32248 tablet_replica.cc:330] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: stopping tablet replica
I20260811 22:24:34.747182 32248 raft_consensus.cc:2229] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:24:34.747650 32248 raft_consensus.cc:2258] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:34.749128 32250 raft_consensus.cc:2229] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:34.749534 32250 raft_consensus.cc:2258] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:34.750000 32248 ts_tablet_manager.cc:1905] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:34.751691 32250 ts_tablet_manager.cc:1905] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:34.753153 32248 ts_tablet_manager.cc:1918] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:34.753274 32248 log.cc:1196] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/9c556305b0a249b2895e064410959917
I20260811 22:24:34.753628 32248 ts_tablet_manager.cc:1939] T 9c556305b0a249b2895e064410959917 P 604e29a52ad843aa88a9016471cff96a: Deleting consensus metadata
I20260811 22:24:34.754807 32250 ts_tablet_manager.cc:1918] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:34.754980 32250 log.cc:1196] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/9c556305b0a249b2895e064410959917
I20260811 22:24:34.755159 31287 catalog_manager.cc:4841] TS 604e29a52ad843aa88a9016471cff96a (127.29.3.67:36853): tablet 9c556305b0a249b2895e064410959917 (table test-table [id=592c2f2bf2f549e38b18a235229ee9ae]) successfully deleted
I20260811 22:24:34.755432 32250 ts_tablet_manager.cc:1939] T 9c556305b0a249b2895e064410959917 P fb5d9e8ac32b469facefef4e9bde8e2f: Deleting consensus metadata
I20260811 22:24:34.756731 31286 catalog_manager.cc:4841] TS fb5d9e8ac32b469facefef4e9bde8e2f (127.29.3.66:45011): tablet 9c556305b0a249b2895e064410959917 (table test-table [id=592c2f2bf2f549e38b18a235229ee9ae]) successfully deleted
W20260811 22:24:34.758116 31283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:41723 (error 108)
I20260811 22:24:34.758456 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31697
I20260811 22:24:34.766073 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:34.765718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:41723 (local address 127.0.0.1:54092)
0811 22:24:34.765882 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:34.765968 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260811 22:24:34.767419 31283 catalog_manager.cc:4584] TS 2e7519c751e1467dbdeca862e109b5bc (127.29.3.65:41723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c556305b0a249b2895e064410959917: Network error: Client connection negotiation failed: client connection to 127.29.3.65:41723: connect: Connection refused (error 111)
I20260811 22:24:34.773217 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31937
I20260811 22:24:34.787997 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 31248
2026-08-11T22:24:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2557 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T22:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:34Z Disabled control of system clock
I20260811 22:24:34.829624 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:43865
--webserver_interface=127.29.3.126
--webserver_port=0
--builtin_ntp_servers=127.29.3.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:43865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:24:34.948853 32256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:34.949154 32256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:34.949191 32256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:34.953346 32256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:34.953429 32256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:34.953444 32256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:34.953459 32256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:34.953473 32256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:34.958640 32256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:43865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:34.959820 32256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:34.961246 32256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:34.970208 32266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.970194 32265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:34.970188 32271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:34.970252 32256 server_base.cc:1034] running on GCE node
I20260811 22:24:34.971062 32256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:34.972270 32256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:34.973441 32256 hybrid_clock.cc:648] HybridClock initialized: now 1786487074973429 us; error 32 us; skew 500 ppm
I20260811 22:24:34.981101 32256 webserver.cc:460] Webserver started at http://127.29.3.126:39699/ using document root <none> and password file <none>
I20260811 22:24:34.981798 32256 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:34.981878 32256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:34.982102 32256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:34.984411 32256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "699c6a525e334a51a9d09b392000a3ba"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "22f00ec64537c82b4b6898c8a1e2784d"
server_key_iv: "97dd45facf02d85e31ca18735c6054f5"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.985002 32256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "699c6a525e334a51a9d09b392000a3ba"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "22f00ec64537c82b4b6898c8a1e2784d"
server_key_iv: "97dd45facf02d85e31ca18735c6054f5"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.989058 32256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:34.992208 32279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:34.993502 32256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:34.993659 32256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "699c6a525e334a51a9d09b392000a3ba"
format_stamp: "Formatted at 2026-08-11 22:24:34 on dist-test-slave-4tdj"
server_key: "22f00ec64537c82b4b6898c8a1e2784d"
server_key_iv: "97dd45facf02d85e31ca18735c6054f5"
server_key_version: "encryptionkey@0"
I20260811 22:24:34.993785 32256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:35.012915 32256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:35.013728 32256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:35.013921 32256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:35.021360 32256 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:43865
I20260811 22:24:35.021382 32378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:43865 every 8 connection(s)
I20260811 22:24:35.022676 32256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:35.026961 32383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:35.027021 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 32256
I20260811 22:24:35.027148 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:35.027474 29709 external_mini_cluster.cc:1428] Setting key 08da24ec6f1de2016142b2e28bc85267
I20260811 22:24:35.034497 32383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: Bootstrap starting.
I20260811 22:24:35.037138 32383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:35.037248 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.028944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.29.3.126:43865)
0811 22:24:35.029621 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.029638 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.029675 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:24:35.030507 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.030530 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.030977 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.031293 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:35.031658 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.032642 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.033845 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.034368 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.034488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.035436 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:35.035442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:35.035455 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:35.035857 (+ 402us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:35.035917 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:35.035939 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:35.036015 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.036172 (+ 157us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.036459 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260811 22:24:35.037986 32383 log.cc:824] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:35.040320 32383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: No bootstrap required, opened a new log
I20260811 22:24:35.044312 32383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "699c6a525e334a51a9d09b392000a3ba" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43865 } }
I20260811 22:24:35.044598 32383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:35.044667 32383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699c6a525e334a51a9d09b392000a3ba, State: Initialized, Role: FOLLOWER
I20260811 22:24:35.045159 32383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "699c6a525e334a51a9d09b392000a3ba" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43865 } }
I20260811 22:24:35.045351 32383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:35.045426 32383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:35.045542 32383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:35.046676 32383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "699c6a525e334a51a9d09b392000a3ba" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43865 } }
I20260811 22:24:35.047055 32383 leader_election.cc:302] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 699c6a525e334a51a9d09b392000a3ba; no voters:
I20260811 22:24:35.047420 32383 leader_election.cc:288] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:35.047523 32390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:35.047770 32390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [term 1 LEADER]: Becoming Leader. State: Replica: 699c6a525e334a51a9d09b392000a3ba, State: Running, Role: LEADER
I20260811 22:24:35.048107 32390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "699c6a525e334a51a9d09b392000a3ba" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43865 } }
I20260811 22:24:35.048743 32383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:35.050026 32392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 699c6a525e334a51a9d09b392000a3ba. Latest consensus state: current_term: 1 leader_uuid: "699c6a525e334a51a9d09b392000a3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "699c6a525e334a51a9d09b392000a3ba" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43865 } } }
I20260811 22:24:35.050175 32392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:35.050607 32391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "699c6a525e334a51a9d09b392000a3ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "699c6a525e334a51a9d09b392000a3ba" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 43865 } } }
I20260811 22:24:35.050719 32391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba [sys.catalog]: This master's current role is: LEADER
W20260811 22:24:35.053787 32418 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:24:35.053867 32418 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:24:35.053996 32407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:35.054828 32407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:35.060626 32407 catalog_manager.cc:1347] Generated new cluster ID: 720b573e68f54b689ce67e5a1622faa4
I20260811 22:24:35.060709 32407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:35.066814 32407 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:35.067770 32407 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:35.077558 32407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: Generated new TSK 0
I20260811 22:24:35.078593 32407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:35.082470 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.65:0
--local_ip_for_outbound_sockets=127.29.3.65
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43865
--builtin_ntp_servers=127.29.3.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 22:24:35.201877 32426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:35.202184 32426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:35.202243 32426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:35.206506 32426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:35.206583 32426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:35.206663 32426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.65
I20260811 22:24:35.211757 32426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.3.65
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.65
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:35.212934 32426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:35.214406 32426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:35.224398 32445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:35.224450 32453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:35.224491 32443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:35.224606 32426 server_base.cc:1034] running on GCE node
I20260811 22:24:35.225136 32426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:35.225970 32426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:35.227159 32426 hybrid_clock.cc:648] HybridClock initialized: now 1786487075227122 us; error 56 us; skew 500 ppm
I20260811 22:24:35.229192 32426 webserver.cc:460] Webserver started at http://127.29.3.65:39309/ using document root <none> and password file <none>
I20260811 22:24:35.229864 32426 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:35.229934 32426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:35.230147 32426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:35.232448 32426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data/instance:
uuid: "5b91887e82964603b7c6edb37dac9b17"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "3915f9eb0fc716edae129893913a797f"
server_key_iv: "d3621bf1b802beda76e2e383501dfc82"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.233053 32426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance:
uuid: "5b91887e82964603b7c6edb37dac9b17"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "3915f9eb0fc716edae129893913a797f"
server_key_iv: "d3621bf1b802beda76e2e383501dfc82"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.237277 32426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:35.240068 32466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.241420 32426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:35.241559 32426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal
uuid: "5b91887e82964603b7c6edb37dac9b17"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "3915f9eb0fc716edae129893913a797f"
server_key_iv: "d3621bf1b802beda76e2e383501dfc82"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.241688 32426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:35.259395 32426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:35.260262 32426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:35.260459 32426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:35.261130 32426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:35.262192 32426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:35.262248 32426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.262326 32426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:35.262357 32426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.271262 32426 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.65:34461
I20260811 22:24:35.271313 32688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.65:34461 every 8 connection(s)
I20260811 22:24:35.272444 32426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/data/info.pb
I20260811 22:24:35.280551 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 32426
I20260811 22:24:35.280685 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-0/wal/instance
I20260811 22:24:35.280968 29709 external_mini_cluster.cc:1428] Setting key 133fd3c125ed3cc78438b2b9bb105355
I20260811 22:24:35.282359 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.274413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.65:51617 (local address 127.29.3.126:43865)
0811 22:24:35.274582 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.274585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.275749 (+ 1164us) server_negotiation.cc:407] Connection header received
0811 22:24:35.277309 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.277312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.277378 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.277455 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:35.278517 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.279085 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.280308 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.280552 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.280650 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.281858 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:35.281861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:35.281863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:35.281875 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:35.281895 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:35.281904 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:35.281962 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.282130 (+ 168us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.282190 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 22:24:35.283167 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.66:0
--local_ip_for_outbound_sockets=127.29.3.66
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43865
--builtin_ntp_servers=127.29.3.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:24:35.283479 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.274867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:43865 (local address 127.29.3.65:51617)
0811 22:24:35.275584 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.275628 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.277056 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.277665 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.277673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.277777 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:35.278348 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.278366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.279231 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.279233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.280166 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.280178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.280738 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.280743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.280920 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.281554 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:35.281556 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:35.281603 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:35.281702 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:35.281981 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:35.281991 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:35.281998 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.282136 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.282615 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":131,"threads_started":1}
I20260811 22:24:35.284789 32689 heartbeater.cc:347] Connected to a master server at 127.29.3.126:43865
I20260811 22:24:35.285033 32689 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:35.285588 32689 heartbeater.cc:511] Master 127.29.3.126:43865 requested a full tablet report, sending...
I20260811 22:24:35.287286 32313 ts_manager.cc:194] Registered new tserver with Master: 5b91887e82964603b7c6edb37dac9b17 (127.29.3.65:34461)
I20260811 22:24:35.288913 32313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.65:51617
W20260811 22:24:35.403622 32693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:35.403961 32693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:35.404002 32693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:35.408141 32693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:35.408210 32693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:35.408268 32693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.66
I20260811 22:24:35.413146 32693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.3.66
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.66
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:35.414256 32693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:35.415746 32693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:35.425710 32707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:35.425715 32710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:35.425813 32693 server_base.cc:1034] running on GCE node
W20260811 22:24:35.425715 32708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:35.426437 32693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:35.427130 32693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:35.428293 32693 hybrid_clock.cc:648] HybridClock initialized: now 1786487075428271 us; error 38 us; skew 500 ppm
I20260811 22:24:35.430349 32693 webserver.cc:460] Webserver started at http://127.29.3.66:39833/ using document root <none> and password file <none>
I20260811 22:24:35.430961 32693 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:35.431034 32693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:35.431243 32693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:35.433409 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data/instance:
uuid: "8e4bed20231344e8bd04f8eb6daced94"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "3ff2c9537c2eea6c876694b7c2b81b89"
server_key_iv: "69e423d9e8bc9de81199e3c4b422c2ab"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.433983 32693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance:
uuid: "8e4bed20231344e8bd04f8eb6daced94"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "3ff2c9537c2eea6c876694b7c2b81b89"
server_key_iv: "69e423d9e8bc9de81199e3c4b422c2ab"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.438040 32693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 22:24:35.440685 32720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.442021 32693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:35.442162 32693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal
uuid: "8e4bed20231344e8bd04f8eb6daced94"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "3ff2c9537c2eea6c876694b7c2b81b89"
server_key_iv: "69e423d9e8bc9de81199e3c4b422c2ab"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.442281 32693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:35.505270 32693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:35.506083 32693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:35.506279 32693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:35.506982 32693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:35.508037 32693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:35.508100 32693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.508159 32693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:35.508185 32693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.517424 32693 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.66:44865
I20260811 22:24:35.517510 462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.66:44865 every 8 connection(s)
I20260811 22:24:35.518754 32693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/data/info.pb
I20260811 22:24:35.522641 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 32693
I20260811 22:24:35.522787 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal/instance
I20260811 22:24:35.523099 29709 external_mini_cluster.cc:1428] Setting key 15d8e3795604c046ad4cbe9de89231a3
I20260811 22:24:35.525429 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.3.67:0
--local_ip_for_outbound_sockets=127.29.3.67
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43865
--builtin_ntp_servers=127.29.3.84:44925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 22:24:35.529531 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.520965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:36215 (local address 127.29.3.126:43865)
0811 22:24:35.521098 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.521102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.522261 (+ 1159us) server_negotiation.cc:407] Connection header received
0811 22:24:35.524150 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.524153 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.524200 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.524287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:35.525494 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.526087 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.527630 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.527860 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.527950 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.529048 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:35.529052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:35.529055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:35.529071 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:35.529090 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:35.529098 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:35.529158 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.529313 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.529383 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:24:35.530568 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.521381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:43865 (local address 127.29.3.66:36215)
0811 22:24:35.522111 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.522158 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.523896 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.524429 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.524436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.524554 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:35.525324 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.525349 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.526220 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.526224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.527507 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.527518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.528008 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.528012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.528190 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.528780 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:35.528781 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:35.528823 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:35.528897 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:35.529168 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:35.529178 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:35.529184 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.529304 (+ 120us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.529618 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":115,"threads_started":1}
I20260811 22:24:35.531738 466 heartbeater.cc:347] Connected to a master server at 127.29.3.126:43865
I20260811 22:24:35.531986 466 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:35.532533 466 heartbeater.cc:511] Master 127.29.3.126:43865 requested a full tablet report, sending...
I20260811 22:24:35.533684 32313 ts_manager.cc:194] Registered new tserver with Master: 8e4bed20231344e8bd04f8eb6daced94 (127.29.3.66:44865)
I20260811 22:24:35.534586 32313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.66:36215
W20260811 22:24:35.650592 475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:35.650913 475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:35.650967 475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:35.655179 475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:35.655241 475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:35.655300 475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.3.67
I20260811 22:24:35.660293 475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:44925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.3.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.3.67
--webserver_port=0
--tserver_master_addrs=127.29.3.126:43865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.3.67
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:35.661520 475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:35.663028 475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:35.672935 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:35.672986 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:35.673139 475 server_base.cc:1034] running on GCE node
W20260811 22:24:35.673295 487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:35.673766 475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:35.674490 475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:35.675665 475 hybrid_clock.cc:648] HybridClock initialized: now 1786487075675630 us; error 60 us; skew 500 ppm
I20260811 22:24:35.677711 475 webserver.cc:460] Webserver started at http://127.29.3.67:40455/ using document root <none> and password file <none>
I20260811 22:24:35.678368 475 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:35.678442 475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:35.678673 475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:35.680871 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data/instance:
uuid: "10866cc0442c44f5b0f7b7acc6b36ead"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "4dd7ac95cd057b17873bb81bb9bfec7e"
server_key_iv: "aa1c3025e8a3de011c38f10508ffe86e"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.681468 475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance:
uuid: "10866cc0442c44f5b0f7b7acc6b36ead"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "4dd7ac95cd057b17873bb81bb9bfec7e"
server_key_iv: "aa1c3025e8a3de011c38f10508ffe86e"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.685817 475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:35.688674 502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.690039 475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:24:35.690164 475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal
uuid: "10866cc0442c44f5b0f7b7acc6b36ead"
format_stamp: "Formatted at 2026-08-11 22:24:35 on dist-test-slave-4tdj"
server_key: "4dd7ac95cd057b17873bb81bb9bfec7e"
server_key_iv: "aa1c3025e8a3de011c38f10508ffe86e"
server_key_version: "encryptionkey@0"
I20260811 22:24:35.690318 475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:35.707362 475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:35.708106 475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:35.708290 475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:35.708942 475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 22:24:35.709949 475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 22:24:35.709997 475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.710057 475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 22:24:35.710084 475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:35.719121 475 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.67:39025
I20260811 22:24:35.719149 699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.67:39025 every 8 connection(s)
I20260811 22:24:35.720144 475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/data/info.pb
I20260811 22:24:35.723840 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 475
I20260811 22:24:35.723937 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal/instance
I20260811 22:24:35.724188 29709 external_mini_cluster.cc:1428] Setting key 67fd86bfe72f513dad1192319395c654
I20260811 22:24:35.730113 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.722126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.67:47175 (local address 127.29.3.126:43865)
0811 22:24:35.722460 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.722465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.723677 (+ 1212us) server_negotiation.cc:407] Connection header received
0811 22:24:35.725291 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.725295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.725356 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.725431 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:35.726412 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.726943 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.728132 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.728378 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.728469 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.729593 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:35.729597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:35.729599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:35.729616 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:35.729636 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:35.729644 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:35.729704 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.729885 (+ 181us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.729956 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260811 22:24:35.731130 702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.722749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:43865 (local address 127.29.3.67:47175)
0811 22:24:35.723511 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.723556 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.725052 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.725617 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.725625 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.725727 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:24:35.726252 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.726266 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.727069 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.727072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.728007 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.728017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.728529 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.728533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.728705 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.729315 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 22:24:35.729317 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:24:35.729363 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 22:24:35.729439 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:24:35.729739 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:24:35.729748 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:24:35.729753 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.729876 (+ 123us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.730203 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":133,"threads_started":1}
I20260811 22:24:35.732323 700 heartbeater.cc:347] Connected to a master server at 127.29.3.126:43865
I20260811 22:24:35.732578 700 heartbeater.cc:464] Registering TS with master...
I20260811 22:24:35.733143 700 heartbeater.cc:511] Master 127.29.3.126:43865 requested a full tablet report, sending...
I20260811 22:24:35.734229 32313 ts_manager.cc:194] Registered new tserver with Master: 10866cc0442c44f5b0f7b7acc6b36ead (127.29.3.67:39025)
I20260811 22:24:35.734836 32313 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.3.67:47175
I20260811 22:24:35.737968 29709 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 22:24:35.743439 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.739948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47492 (local address 127.29.3.126:43865)
0811 22:24:35.740081 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.740086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.740241 (+ 155us) server_negotiation.cc:407] Connection header received
0811 22:24:35.740367 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.740369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.740411 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.740467 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:35.740678 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.741200 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.741992 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.742197 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.742272 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.743069 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:35.743072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:35.743074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:35.743086 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:35.743102 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:35.743109 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:35.743164 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.743238 (+ 74us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.743290 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:24:35.746496 32313 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 22:24:35.749820 32313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 22:24:35.765564 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.760415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37512 (local address 127.29.3.66:44865)
0811 22:24:35.760821 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.760832 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.760855 (+ 23us) server_negotiation.cc:407] Connection header received
0811 22:24:35.760921 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.760925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.761076 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.761637 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:35.761778 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.762564 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.764218 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.764837 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.764939 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.765164 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.765284 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.765395 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":131,"threads_started":1}
I20260811 22:24:35.765564 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.759518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.66:44865 (local address 127.0.0.1:37512)
0811 22:24:35.760422 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.760436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.760529 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.761263 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.761269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.761302 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:35.761472 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.761480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.762683 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.762687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.764063 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.764076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.764966 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.764969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.765081 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.765120 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.765252 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.765395 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":28,"spinlock_wait_cycles":3456,"thread_start_us":130,"threads_started":1}
I20260811 22:24:35.765947 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.759344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34461 (local address 127.0.0.1:54954)
0811 22:24:35.759953 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.760013 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.760154 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.761417 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.761420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.761435 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:35.761541 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.761547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.763094 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.763097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.764279 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.764289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.765586 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.765589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.765666 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.765686 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.765748 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.765814 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":102,"thread_start_us":74,"threads_started":1}
I20260811 22:24:35.766009 714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.760119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:39025 (local address 127.0.0.1:33818)
0811 22:24:35.760473 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.760485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.760572 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.761264 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.761271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.761307 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:35.761434 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.761440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.763073 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.763077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.764251 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.764260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.765342 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.765346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.765431 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.765453 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.765501 (+ 48us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.765575 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":40,"spinlock_wait_cycles":3840,"thread_start_us":102,"threads_started":1}
I20260811 22:24:35.766171 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.760282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.29.3.67:39025)
0811 22:24:35.760762 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.760773 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.760798 (+ 25us) server_negotiation.cc:407] Connection header received
0811 22:24:35.760876 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.760881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.761101 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.761563 (+ 462us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:35.761700 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.762938 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.764370 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.765188 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.765662 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.765854 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.765902 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.765999 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":126,"threads_started":1}
I20260811 22:24:35.766449 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.759529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54954 (local address 127.29.3.65:34461)
0811 22:24:35.760124 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.760132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.760160 (+ 28us) server_negotiation.cc:407] Connection header received
0811 22:24:35.760820 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.760827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.761076 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.761248 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:35.761677 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.762963 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.764421 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.765188 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.765310 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.765515 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.766190 (+ 675us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.766326 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":122,"threads_started":1}
I20260811 22:24:35.769156 338 tablet_service.cc:1467] Processing CreateTablet for tablet 447be162a68142e1acde11ec67b70022 (DEFAULT_TABLE table=test-table [id=c0df280e331747fa89aba4fe7885b31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:35.769305 583 tablet_service.cc:1467] Processing CreateTablet for tablet 447be162a68142e1acde11ec67b70022 (DEFAULT_TABLE table=test-table [id=c0df280e331747fa89aba4fe7885b31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:35.769623 32561 tablet_service.cc:1467] Processing CreateTablet for tablet 447be162a68142e1acde11ec67b70022 (DEFAULT_TABLE table=test-table [id=c0df280e331747fa89aba4fe7885b31f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 22:24:35.770169 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447be162a68142e1acde11ec67b70022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:35.770442 583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447be162a68142e1acde11ec67b70022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:35.770516 32561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 447be162a68142e1acde11ec67b70022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:35.777036 723 tablet_bootstrap.cc:492] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Bootstrap starting.
I20260811 22:24:35.778432 728 tablet_bootstrap.cc:492] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Bootstrap starting.
I20260811 22:24:35.778600 724 tablet_bootstrap.cc:492] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17: Bootstrap starting.
I20260811 22:24:35.779573 723 tablet_bootstrap.cc:654] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:35.780507 723 log.cc:824] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:35.780726 728 tablet_bootstrap.cc:654] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:35.780725 724 tablet_bootstrap.cc:654] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:35.781638 728 log.cc:824] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:35.782058 724 log.cc:824] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:35.782662 723 tablet_bootstrap.cc:492] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: No bootstrap required, opened a new log
I20260811 22:24:35.782882 723 ts_tablet_manager.cc:1397] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 22:24:35.783788 724 tablet_bootstrap.cc:492] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17: No bootstrap required, opened a new log
I20260811 22:24:35.783811 728 tablet_bootstrap.cc:492] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: No bootstrap required, opened a new log
I20260811 22:24:35.783979 724 ts_tablet_manager.cc:1397] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260811 22:24:35.783991 728 ts_tablet_manager.cc:1397] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 22:24:35.786909 724 raft_consensus.cc:348] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.787171 724 raft_consensus.cc:374] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:35.787238 724 raft_consensus.cc:729] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b91887e82964603b7c6edb37dac9b17, State: Initialized, Role: FOLLOWER
I20260811 22:24:35.787691 724 consensus_queue.cc:260] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.788326 32689 heartbeater.cc:503] Master 127.29.3.126:43865 was elected leader, sending a full tablet report...
I20260811 22:24:35.788198 728 raft_consensus.cc:348] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.788466 728 raft_consensus.cc:374] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:35.788511 728 raft_consensus.cc:729] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10866cc0442c44f5b0f7b7acc6b36ead, State: Initialized, Role: FOLLOWER
I20260811 22:24:35.788445 723 raft_consensus.cc:348] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.788687 723 raft_consensus.cc:374] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:35.788738 723 raft_consensus.cc:729] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4bed20231344e8bd04f8eb6daced94, State: Initialized, Role: FOLLOWER
I20260811 22:24:35.788763 724 ts_tablet_manager.cc:1428] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260811 22:24:35.788973 728 consensus_queue.cc:260] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.789215 723 consensus_queue.cc:260] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.789559 739 raft_consensus.cc:482] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 22:24:35.790041 728 ts_tablet_manager.cc:1428] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 22:24:35.790071 723 ts_tablet_manager.cc:1428] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260811 22:24:35.790001 739 raft_consensus.cc:504] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.790351 700 heartbeater.cc:503] Master 127.29.3.126:43865 was elected leader, sending a full tablet report...
I20260811 22:24:35.790433 466 heartbeater.cc:503] Master 127.29.3.126:43865 was elected leader, sending a full tablet report...
I20260811 22:24:35.792778 739 leader_election.cc:288] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b91887e82964603b7c6edb37dac9b17 (127.29.3.65:34461), 10866cc0442c44f5b0f7b7acc6b36ead (127.29.3.67:39025)
I20260811 22:24:35.796429 471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.792143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34461 (local address 127.29.3.66:37171)
0811 22:24:35.792295 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.792313 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.792397 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.793003 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.793006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.793026 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:35.793162 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.793172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.794142 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.794145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.795177 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.795189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.796072 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.796074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.796165 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.796188 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.796259 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.796321 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 22:24:35.796486 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.792621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:37171 (local address 127.29.3.65:34461)
0811 22:24:35.792741 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.792745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.792759 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:24:35.792806 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.792809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.792854 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.792930 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:35.793301 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.794019 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.795301 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.795939 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.796044 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.796108 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.796287 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.796358 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:24:35.797155 32598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447be162a68142e1acde11ec67b70022" candidate_uuid: "8e4bed20231344e8bd04f8eb6daced94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b91887e82964603b7c6edb37dac9b17" is_pre_election: true
I20260811 22:24:35.797570 32598 raft_consensus.cc:2454] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4bed20231344e8bd04f8eb6daced94 in term 0.
I20260811 22:24:35.797781 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.792645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.3.66:41605 (local address 127.29.3.67:39025)
0811 22:24:35.792788 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.792793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.793758 (+ 965us) server_negotiation.cc:407] Connection header received
0811 22:24:35.793915 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.793918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.793958 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.793999 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 22:24:35.794484 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.795210 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.796496 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.797055 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.797186 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.797252 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.797571 (+ 319us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.797656 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 22:24:35.797791 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.792548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.67:39025 (local address 127.29.3.66:41605)
0811 22:24:35.793679 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.793693 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:24:35.793817 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:24:35.794129 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:24:35.794132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:24:35.794148 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 22:24:35.794258 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.794263 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.795312 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.795315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.796359 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:24:35.796370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.797263 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.797267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:24:35.797365 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.797398 (+ 33us) client_negotiation.cc:769] Sending connection context
0811 22:24:35.797544 (+ 146us) client_negotiation.cc:240] Negotiation successful
0811 22:24:35.797616 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"thread_start_us":116,"threads_started":1}
I20260811 22:24:35.798139 32727 leader_election.cc:302] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b91887e82964603b7c6edb37dac9b17, 8e4bed20231344e8bd04f8eb6daced94; no voters:
I20260811 22:24:35.798501 739 raft_consensus.cc:2781] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 22:24:35.798506 614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447be162a68142e1acde11ec67b70022" candidate_uuid: "8e4bed20231344e8bd04f8eb6daced94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" is_pre_election: true
I20260811 22:24:35.798626 739 raft_consensus.cc:482] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 22:24:35.798689 739 raft_consensus.cc:3037] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:35.798792 614 raft_consensus.cc:2454] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4bed20231344e8bd04f8eb6daced94 in term 0.
I20260811 22:24:35.799929 739 raft_consensus.cc:504] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.800341 739 leader_election.cc:288] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [CANDIDATE]: Term 1 election: Requested vote from peers 5b91887e82964603b7c6edb37dac9b17 (127.29.3.65:34461), 10866cc0442c44f5b0f7b7acc6b36ead (127.29.3.67:39025)
I20260811 22:24:35.800721 614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447be162a68142e1acde11ec67b70022" candidate_uuid: "8e4bed20231344e8bd04f8eb6daced94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "10866cc0442c44f5b0f7b7acc6b36ead"
I20260811 22:24:35.800855 614 raft_consensus.cc:3037] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:35.802239 614 raft_consensus.cc:2454] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4bed20231344e8bd04f8eb6daced94 in term 1.
I20260811 22:24:35.802448 32598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "447be162a68142e1acde11ec67b70022" candidate_uuid: "8e4bed20231344e8bd04f8eb6daced94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b91887e82964603b7c6edb37dac9b17"
I20260811 22:24:35.802618 32730 leader_election.cc:302] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10866cc0442c44f5b0f7b7acc6b36ead, 8e4bed20231344e8bd04f8eb6daced94; no voters:
I20260811 22:24:35.802603 32598 raft_consensus.cc:3037] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:35.802788 739 raft_consensus.cc:2781] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:35.802892 739 raft_consensus.cc:686] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 1 LEADER]: Becoming Leader. State: Replica: 8e4bed20231344e8bd04f8eb6daced94, State: Running, Role: LEADER
I20260811 22:24:35.803261 739 consensus_queue.cc:237] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } }
I20260811 22:24:35.803783 32598 raft_consensus.cc:2454] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4bed20231344e8bd04f8eb6daced94 in term 1.
I20260811 22:24:35.806779 32313 catalog_manager.cc:5495] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e4bed20231344e8bd04f8eb6daced94 (127.29.3.66). New cstate: current_term: 1 leader_uuid: "8e4bed20231344e8bd04f8eb6daced94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e4bed20231344e8bd04f8eb6daced94" member_type: VOTER last_known_addr { host: "127.29.3.66" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 } health_report { overall_health: UNKNOWN } } }
I20260811 22:24:35.820009 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.816783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.29.3.66:44865)
0811 22:24:35.816916 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:35.816919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:35.816931 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:24:35.817010 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:35.817011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:35.817067 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:35.817156 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:24:35.817344 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.817856 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.818710 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:35.818921 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:35.818998 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:35.819238 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:24:35.819662 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:24:35.819759 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:35.819815 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 22:24:35.819866 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 22:24:35.826128 614 raft_consensus.cc:1261] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 1 FOLLOWER]: Refusing update from remote peer 8e4bed20231344e8bd04f8eb6daced94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:35.826128 32598 raft_consensus.cc:1261] T 447be162a68142e1acde11ec67b70022 P 5b91887e82964603b7c6edb37dac9b17 [term 1 FOLLOWER]: Refusing update from remote peer 8e4bed20231344e8bd04f8eb6daced94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:24:35.826958 741 consensus_queue.cc:1035] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b91887e82964603b7c6edb37dac9b17" member_type: VOTER last_known_addr { host: "127.29.3.65" port: 34461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:35.827150 739 consensus_queue.cc:1035] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10866cc0442c44f5b0f7b7acc6b36ead" member_type: VOTER last_known_addr { host: "127.29.3.67" port: 39025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:24:35.833298 748 mvcc.cc:205] Tried to move back new op lower bound from 7317451062576017408 to 7317451062493655040. Current Snapshot: MvccSnapshot[applied={T|T < 7317451062576017408}]
I20260811 22:24:35.835623 752 mvcc.cc:205] Tried to move back new op lower bound from 7317451062576017408 to 7317451062493655040. Current Snapshot: MvccSnapshot[applied={T|T < 7317451062576017408}]
I20260811 22:24:35.835893 749 mvcc.cc:205] Tried to move back new op lower bound from 7317451062576017408 to 7317451062493655040. Current Snapshot: MvccSnapshot[applied={T|T < 7317451062576017408}]
I20260811 22:24:35.843165 32313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:35.843329 32313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 22:24:35.845820 32313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 699c6a525e334a51a9d09b392000a3ba: Sending DeleteTablet for 3 replicas of tablet 447be162a68142e1acde11ec67b70022
I20260811 22:24:35.846861 338 tablet_service.cc:1514] Processing DeleteTablet for tablet 447be162a68142e1acde11ec67b70022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:35 UTC) from {username='slave'} at 127.0.0.1:37512
I20260811 22:24:35.846863 583 tablet_service.cc:1514] Processing DeleteTablet for tablet 447be162a68142e1acde11ec67b70022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:35 UTC) from {username='slave'} at 127.0.0.1:33818
I20260811 22:24:35.847465 762 tablet_replica.cc:330] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: stopping tablet replica
I20260811 22:24:35.847592 32561 tablet_service.cc:1514] Processing DeleteTablet for tablet 447be162a68142e1acde11ec67b70022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 22:24:35 UTC) from {username='slave'} at 127.0.0.1:54954
I20260811 22:24:35.847998 762 raft_consensus.cc:2229] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 1 LEADER]: Raft consensus shutting down.
I20260811 22:24:35.848196 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 32426
I20260811 22:24:35.848454 762 raft_consensus.cc:2258] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 22:24:35.852856 762 ts_tablet_manager.cc:1905] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:35.855778 762 ts_tablet_manager.cc:1918] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:35.855886 762 log.cc:1196] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-1/wal/wals/447be162a68142e1acde11ec67b70022
I20260811 22:24:35.856201 762 ts_tablet_manager.cc:1939] T 447be162a68142e1acde11ec67b70022 P 8e4bed20231344e8bd04f8eb6daced94: Deleting consensus metadata
I20260811 22:24:35.857566 763 tablet_replica.cc:330] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: stopping tablet replica
I20260811 22:24:35.857596 32285 catalog_manager.cc:4841] TS 8e4bed20231344e8bd04f8eb6daced94 (127.29.3.66:44865): tablet 447be162a68142e1acde11ec67b70022 (table test-table [id=c0df280e331747fa89aba4fe7885b31f]) successfully deleted
I20260811 22:24:35.857937 763 raft_consensus.cc:2229] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 22:24:35.858311 763 raft_consensus.cc:2258] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 22:24:35.863519 32282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.3.65:34461 (error 108)
I20260811 22:24:35.863741 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 32693
I20260811 22:24:35.864014 763 ts_tablet_manager.cc:1905] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 22:24:35.866573 763 ts_tablet_manager.cc:1918] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 22:24:35.866684 763 log.cc:1196] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Deleting WAL directory at /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786486810933177-29709-0/minicluster-data/ts-2/wal/wals/447be162a68142e1acde11ec67b70022
I20260811 22:24:35.866962 763 ts_tablet_manager.cc:1939] T 447be162a68142e1acde11ec67b70022 P 10866cc0442c44f5b0f7b7acc6b36ead: Deleting consensus metadata
I20260811 22:24:35.867763 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:35.867388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.65:34461 (local address 127.0.0.1:54960)
0811 22:24:35.867542 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 22:24:35.867649 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260811 22:24:35.869664 32285 catalog_manager.cc:4841] TS 10866cc0442c44f5b0f7b7acc6b36ead (127.29.3.67:39025): tablet 447be162a68142e1acde11ec67b70022 (table test-table [id=c0df280e331747fa89aba4fe7885b31f]) successfully deleted
W20260811 22:24:35.872151 32282 catalog_manager.cc:4584] TS 5b91887e82964603b7c6edb37dac9b17 (127.29.3.65:34461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 447be162a68142e1acde11ec67b70022: Network error: Client connection negotiation failed: client connection to 127.29.3.65:34461: connect: Connection refused (error 111)
I20260811 22:24:35.877799 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 475
I20260811 22:24:35.892297 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 32256
2026-08-11T22:24:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1106 ms)
[----------] 6 tests from EncryptionPolicyTest (11552 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:35 dist-test-slave-4tdj krb5kdc[775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:35 dist-test-slave-4tdj krb5kdc[775](info): set up 2 sockets
Aug 11 22:24:35 dist-test-slave-4tdj krb5kdc[775](info): commencing operation
krb5kdc: starting...
W20260811 22:24:37.112828 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:37 dist-test-slave-4tdj krb5kdc[775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:37.245751 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:37.364017 795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:37.364372 795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:37.364421 795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:37.369699 795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:37.369792 795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:37.369819 795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:37.369853 795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:37.369882 795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:37.376978 795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42221
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:37.378420 795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:37.379693 795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:37.393698 805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:37.393913 810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:37.393743 795 server_base.cc:1034] running on GCE node
W20260811 22:24:37.393700 807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:37.394884 795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:37.396314 795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:37.397485 795 hybrid_clock.cc:648] HybridClock initialized: now 1786487077397458 us; error 49 us; skew 500 ppm
Aug 11 22:24:37 dist-test-slave-4tdj krb5kdc[775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487077, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:37.400797 795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:37.401804 795 webserver.cc:460] Webserver started at http://127.0.0.1:44261/ using document root <none> and password file <none>
I20260811 22:24:37.402428 795 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:37.402504 795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:37.402741 795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:37.404897 795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "21a55a8a8e064ee3be26343f02cd2219"
format_stamp: "Formatted at 2026-08-11 22:24:37 on dist-test-slave-4tdj"
server_key: "4540e64b2f2dcd6d0d2b4bcb05c7c1d7"
server_key_iv: "bcc4889cd64c7c9c0d568de073049b72"
server_key_version: "encryptionkey@0"
I20260811 22:24:37.405544 795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "21a55a8a8e064ee3be26343f02cd2219"
format_stamp: "Formatted at 2026-08-11 22:24:37 on dist-test-slave-4tdj"
server_key: "4540e64b2f2dcd6d0d2b4bcb05c7c1d7"
server_key_iv: "bcc4889cd64c7c9c0d568de073049b72"
server_key_version: "encryptionkey@0"
I20260811 22:24:37.409814 795 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:37.412407 820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:37.413719 795 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:24:37.413848 795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "21a55a8a8e064ee3be26343f02cd2219"
format_stamp: "Formatted at 2026-08-11 22:24:37 on dist-test-slave-4tdj"
server_key: "4540e64b2f2dcd6d0d2b4bcb05c7c1d7"
server_key_iv: "bcc4889cd64c7c9c0d568de073049b72"
server_key_version: "encryptionkey@0"
I20260811 22:24:37.413957 795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:37.425757 795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:37.429157 795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:37.429374 795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:37.437086 795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42221
I20260811 22:24:37.437120 912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42221 every 8 connection(s)
I20260811 22:24:37.438131 795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:37.441223 914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:37.443524 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 795
I20260811 22:24:37.443674 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:37.443979 29709 external_mini_cluster.cc:1428] Setting key 6f6acc610507e747270161e12fedebfd
I20260811 22:24:37.447908 914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219: Bootstrap starting.
I20260811 22:24:37.450367 914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:37.451148 914 log.cc:824] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:24:37 dist-test-slave-4tdj krb5kdc[775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:24:37.453436 914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219: No bootstrap required, opened a new log
I20260811 22:24:37.457206 914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a55a8a8e064ee3be26343f02cd2219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260811 22:24:37.457485 914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:37.457544 914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a55a8a8e064ee3be26343f02cd2219, State: Initialized, Role: FOLLOWER
I20260811 22:24:37.458045 914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a55a8a8e064ee3be26343f02cd2219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260811 22:24:37.458238 914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:37.458333 914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:37.458441 914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:37.458743 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:37.445609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39749 (local address 127.0.0.1:42221)
0811 22:24:37.446120 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:37.446135 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:37.446174 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:24:37.446946 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:37.446965 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:37.447358 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:37.447691 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:37.448561 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:37.449568 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:37.450461 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:37.450981 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:37.451149 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:37.453927 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:37.453933 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:37.453948 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:37.453977 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:37.455571 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:37.456095 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:37.456100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:37.456105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:37.456231 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:37.456548 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:37.456551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:37.456552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:37.456964 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:37.457206 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:37.457487 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 22:24:37.457735 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":148,"threads_started":1}
I20260811 22:24:37.459651 914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a55a8a8e064ee3be26343f02cd2219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260811 22:24:37.460017 914 leader_election.cc:302] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21a55a8a8e064ee3be26343f02cd2219; no voters:
I20260811 22:24:37.460352 914 leader_election.cc:288] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:37.460466 921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:37.460700 921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [term 1 LEADER]: Becoming Leader. State: Replica: 21a55a8a8e064ee3be26343f02cd2219, State: Running, Role: LEADER
I20260811 22:24:37.461048 921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a55a8a8e064ee3be26343f02cd2219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } }
I20260811 22:24:37.461751 914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:37.462260 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:37.464797 922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21a55a8a8e064ee3be26343f02cd2219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a55a8a8e064ee3be26343f02cd2219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } } }
I20260811 22:24:37.464963 922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:37.465291 923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21a55a8a8e064ee3be26343f02cd2219. Latest consensus state: current_term: 1 leader_uuid: "21a55a8a8e064ee3be26343f02cd2219" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21a55a8a8e064ee3be26343f02cd2219" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42221 } } }
I20260811 22:24:37.465392 923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219 [sys.catalog]: This master's current role is: LEADER
W20260811 22:24:37.468611 955 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:24:37.468746 955 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:24:37.468853 952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:37.469749 952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:37.474931 919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:37.465170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46121 (local address 127.0.0.1:42221)
0811 22:24:37.465318 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:37.465323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:37.466154 (+ 831us) server_negotiation.cc:407] Connection header received
0811 22:24:37.466332 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:37.466336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:37.466387 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:37.466476 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:37.467316 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:37.468059 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:37.468918 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:37.469239 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:37.469347 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:37.471185 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:37.471189 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:37.471192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:37.471220 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:37.472901 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:37.473481 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:37.473485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:37.473487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:37.473581 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:37.473938 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:37.473941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:37.473943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:37.474168 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:37.474361 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:37.474660 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 22:24:37.474768 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 22:24:37.476799 952 catalog_manager.cc:1347] Generated new cluster ID: 0b2762878cab409697acadbb4b92ae0b
I20260811 22:24:37.476872 952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:37.485960 952 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:37.487023 952 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:37.492378 952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21a55a8a8e064ee3be26343f02cd2219: Generated new TSK 0
I20260811 22:24:37.493089 952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:37.500917 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 795
W20260811 22:24:37.515954 769 connection.cc:535] client connection to 127.0.0.1:42221 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42221): Connection reset by peer (error 104)
2026-08-11T22:24:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1602 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:37 dist-test-slave-4tdj krb5kdc[965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:37 dist-test-slave-4tdj krb5kdc[965](info): set up 2 sockets
Aug 11 22:24:37 dist-test-slave-4tdj krb5kdc[965](info): commencing operation
krb5kdc: starting...
W20260811 22:24:38.702970 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:38 dist-test-slave-4tdj krb5kdc[965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:38.838559 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:38.957211 996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:38.957525 996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:38.957573 996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:38.961768 996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:38.961858 996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:38.961875 996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:38.961892 996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:38.961906 996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:38.967604 996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:38.968787 996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:38.969869 996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:38.978826 1006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:38.978820 1010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:38.978820 1005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:38.978960 996 server_base.cc:1034] running on GCE node
I20260811 22:24:38.979815 996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:38.980995 996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:38.982191 996 hybrid_clock.cc:648] HybridClock initialized: now 1786487078982167 us; error 31 us; skew 500 ppm
Aug 11 22:24:38 dist-test-slave-4tdj krb5kdc[965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:38.985237 996 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:38.986272 996 webserver.cc:460] Webserver started at http://127.0.0.1:39511/ using document root <none> and password file <none>
I20260811 22:24:38.986904 996 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:38.986977 996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:38.987185 996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:38.989341 996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "8d62e61c7951495a9d50b8af06e6b800"
format_stamp: "Formatted at 2026-08-11 22:24:38 on dist-test-slave-4tdj"
server_key: "80c722ee1ae0b14f3e323990f4e32083"
server_key_iv: "dc00cf622316ba306c61e4535295651b"
server_key_version: "encryptionkey@0"
I20260811 22:24:38.990039 996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "8d62e61c7951495a9d50b8af06e6b800"
format_stamp: "Formatted at 2026-08-11 22:24:38 on dist-test-slave-4tdj"
server_key: "80c722ee1ae0b14f3e323990f4e32083"
server_key_iv: "dc00cf622316ba306c61e4535295651b"
server_key_version: "encryptionkey@0"
I20260811 22:24:38.994362 996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:24:38.997884 1026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:38.999737 996 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260811 22:24:38.999991 996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "8d62e61c7951495a9d50b8af06e6b800"
format_stamp: "Formatted at 2026-08-11 22:24:38 on dist-test-slave-4tdj"
server_key: "80c722ee1ae0b14f3e323990f4e32083"
server_key_iv: "dc00cf622316ba306c61e4535295651b"
server_key_version: "encryptionkey@0"
I20260811 22:24:39.000145 996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:39.031314 996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:39.034693 996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:39.034883 996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:39.041926 996 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38341
I20260811 22:24:39.041941 1118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38341 every 8 connection(s)
I20260811 22:24:39.042938 996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:39.045898 1119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:39.046595 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 996
I20260811 22:24:39.046741 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:39.047055 29709 external_mini_cluster.cc:1428] Setting key aaed08c430ca9b65141813badec90aa9
I20260811 22:24:39.052273 1119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800: Bootstrap starting.
I20260811 22:24:39.054638 1119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:39.055442 1119 log.cc:824] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:39.056183 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:39.048577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58193 (local address 127.0.0.1:38341)
0811 22:24:39.049097 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:39.049112 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:39.049149 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:24:39.049830 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:39.049849 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:39.050197 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:39.050568 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:39.050963 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:39.051920 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:39.052816 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:39.053310 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:39.053458 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:39.054326 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:39.054333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:39.054351 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:39.054760 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:39.054830 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:39.054856 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:39.054974 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:39.055156 (+ 182us) server_negotiation.cc:299] Negotiation successful
0811 22:24:39.055301 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":128,"threads_started":1}
I20260811 22:24:39.057744 1119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800: No bootstrap required, opened a new log
I20260811 22:24:39.059397 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:39.062000 1119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d62e61c7951495a9d50b8af06e6b800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38341 } }
I20260811 22:24:39.062278 1119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:39.062359 1119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d62e61c7951495a9d50b8af06e6b800, State: Initialized, Role: FOLLOWER
I20260811 22:24:39.062840 1119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d62e61c7951495a9d50b8af06e6b800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38341 } }
I20260811 22:24:39.062978 1119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:39.063046 1119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:39.063148 1119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:39.064113 1119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d62e61c7951495a9d50b8af06e6b800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38341 } }
I20260811 22:24:39.064456 1119 leader_election.cc:302] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d62e61c7951495a9d50b8af06e6b800; no voters:
I20260811 22:24:39.064783 1119 leader_election.cc:288] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:39.064922 1138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:39.065124 1138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [term 1 LEADER]: Becoming Leader. State: Replica: 8d62e61c7951495a9d50b8af06e6b800, State: Running, Role: LEADER
I20260811 22:24:39.065445 1138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d62e61c7951495a9d50b8af06e6b800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38341 } }
I20260811 22:24:39.065619 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:39.061416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49215 (local address 127.0.0.1:38341)
0811 22:24:39.061528 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:39.061533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:39.061704 (+ 171us) server_negotiation.cc:407] Connection header received
0811 22:24:39.061860 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:39.061863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:39.061904 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:39.061966 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:39.062235 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:39.062866 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:39.063755 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:39.064009 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:39.064098 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:39.064903 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:39.064907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:39.064909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:39.064923 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:39.064950 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:39.064961 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:39.065036 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:39.065446 (+ 410us) server_negotiation.cc:299] Negotiation successful
0811 22:24:39.065483 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:24:39.066206 1119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:39.067425 1139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d62e61c7951495a9d50b8af06e6b800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d62e61c7951495a9d50b8af06e6b800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38341 } } }
I20260811 22:24:39.067559 1139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:39.067994 1140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d62e61c7951495a9d50b8af06e6b800. Latest consensus state: current_term: 1 leader_uuid: "8d62e61c7951495a9d50b8af06e6b800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d62e61c7951495a9d50b8af06e6b800" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38341 } } }
I20260811 22:24:39.068115 1140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800 [sys.catalog]: This master's current role is: LEADER
W20260811 22:24:39.070942 1164 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:24:39.071020 1164 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:24:39.071108 1156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:39.071894 1156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:39.077834 1156 catalog_manager.cc:1347] Generated new cluster ID: 2ab85f92c7da4c09b319f65a190f391c
I20260811 22:24:39.077929 1156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:39.094137 1156 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:39.095168 1156 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:39.102559 1156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d62e61c7951495a9d50b8af06e6b800: Generated new TSK 0
I20260811 22:24:39.103294 1156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:39.117003 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 996
W20260811 22:24:39.140312 961 connection.cc:535] client connection to 127.0.0.1:38341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38341): Connection reset by peer (error 104)
2026-08-11T22:24:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1624 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:39 dist-test-slave-4tdj krb5kdc[1170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:39 dist-test-slave-4tdj krb5kdc[1170](info): set up 2 sockets
Aug 11 22:24:39 dist-test-slave-4tdj krb5kdc[1170](info): commencing operation
krb5kdc: starting...
W20260811 22:24:40.334515 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:40 dist-test-slave-4tdj krb5kdc[1170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:40.469614 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:40.588057 1199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:40.588387 1199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:40.588449 1199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:40.592592 1199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:40.592681 1199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:40.592711 1199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:40.592733 1199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:40.592754 1199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:40.598537 1199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36133
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:40.599799 1199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:40.600924 1199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:40.609875 1213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:40.609875 1209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:40.610131 1206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:40.609956 1199 server_base.cc:1034] running on GCE node
I20260811 22:24:40.610822 1199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:40.612025 1199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:40.613188 1199 hybrid_clock.cc:648] HybridClock initialized: now 1786487080613171 us; error 32 us; skew 500 ppm
Aug 11 22:24:40 dist-test-slave-4tdj krb5kdc[1170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:40.619678 1199 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:40.620684 1199 webserver.cc:460] Webserver started at http://127.0.0.1:39801/ using document root <none> and password file <none>
I20260811 22:24:40.621279 1199 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:40.621342 1199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:40.621537 1199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:40.623762 1199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "54888c3e679f408e9e0b5618010cad2f"
format_stamp: "Formatted at 2026-08-11 22:24:40 on dist-test-slave-4tdj"
server_key: "730a550ee44583c6dcb6b2f83e40e5c8"
server_key_iv: "cea9ba6b1f632f9a2d53d9c0173841b8"
server_key_version: "encryptionkey@0"
I20260811 22:24:40.624331 1199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "54888c3e679f408e9e0b5618010cad2f"
format_stamp: "Formatted at 2026-08-11 22:24:40 on dist-test-slave-4tdj"
server_key: "730a550ee44583c6dcb6b2f83e40e5c8"
server_key_iv: "cea9ba6b1f632f9a2d53d9c0173841b8"
server_key_version: "encryptionkey@0"
I20260811 22:24:40.628331 1199 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:40.630941 1228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:40.632278 1199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:40.632412 1199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "54888c3e679f408e9e0b5618010cad2f"
format_stamp: "Formatted at 2026-08-11 22:24:40 on dist-test-slave-4tdj"
server_key: "730a550ee44583c6dcb6b2f83e40e5c8"
server_key_iv: "cea9ba6b1f632f9a2d53d9c0173841b8"
server_key_version: "encryptionkey@0"
I20260811 22:24:40.632524 1199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:40.637389 1199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:40.640820 1199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:40.641021 1199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:40.648308 1199 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36133
I20260811 22:24:40.648352 1332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36133 every 8 connection(s)
I20260811 22:24:40.649290 1199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:40.652444 1333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:40.657433 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1199
I20260811 22:24:40.657577 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:40.657832 29709 external_mini_cluster.cc:1428] Setting key 59207f24ce6fa9ecf69c98d2146acfe2
I20260811 22:24:40.658416 1333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f: Bootstrap starting.
I20260811 22:24:40.660887 1333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:40.661748 1333 log.cc:824] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:40.663288 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:40.659340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33743 (local address 127.0.0.1:36133)
0811 22:24:40.659793 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:40.659810 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:40.659846 (+ 36us) server_negotiation.cc:407] Connection header received
0811 22:24:40.660490 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:40.660508 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:40.660819 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:40.661189 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:40.661986 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:40.661991 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:40.662003 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:40.662013 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:40.662081 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:40.662102 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:40.662179 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:40.662386 (+ 207us) server_negotiation.cc:299] Negotiation successful
0811 22:24:40.662550 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20260811 22:24:40.663829 1333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f: No bootstrap required, opened a new log
I20260811 22:24:40.666353 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:40.667997 1333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54888c3e679f408e9e0b5618010cad2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36133 } }
I20260811 22:24:40.668367 1333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:40.668485 1333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54888c3e679f408e9e0b5618010cad2f, State: Initialized, Role: FOLLOWER
I20260811 22:24:40.668956 1333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54888c3e679f408e9e0b5618010cad2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36133 } }
I20260811 22:24:40.669099 1333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:40.669168 1333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:40.669268 1333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:40.669951 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:40.668358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60839 (local address 127.0.0.1:36133)
0811 22:24:40.668477 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:40.668482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:40.668637 (+ 155us) server_negotiation.cc:407] Connection header received
0811 22:24:40.668772 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:40.668775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:40.668810 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:40.668873 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:40.669584 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:40.669588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:40.669590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:40.669602 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:40.669630 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:40.669641 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:40.669721 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:40.669789 (+ 68us) server_negotiation.cc:299] Negotiation successful
0811 22:24:40.669831 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 22:24:40.670418 1333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54888c3e679f408e9e0b5618010cad2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36133 } }
I20260811 22:24:40.670806 1333 leader_election.cc:302] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54888c3e679f408e9e0b5618010cad2f; no voters:
I20260811 22:24:40.671200 1333 leader_election.cc:288] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:40.671291 1349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:40.671500 1349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [term 1 LEADER]: Becoming Leader. State: Replica: 54888c3e679f408e9e0b5618010cad2f, State: Running, Role: LEADER
I20260811 22:24:40.671852 1349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54888c3e679f408e9e0b5618010cad2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36133 } }
I20260811 22:24:40.672465 1333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:40.673498 1350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54888c3e679f408e9e0b5618010cad2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54888c3e679f408e9e0b5618010cad2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36133 } } }
I20260811 22:24:40.673633 1350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:40.673889 1354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54888c3e679f408e9e0b5618010cad2f. Latest consensus state: current_term: 1 leader_uuid: "54888c3e679f408e9e0b5618010cad2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54888c3e679f408e9e0b5618010cad2f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36133 } } }
I20260811 22:24:40.673966 1354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:40.674019 1367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:40.677490 1367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:40.684055 1367 catalog_manager.cc:1347] Generated new cluster ID: 21d1e9bf695d4f4ab143c437458d19bd
I20260811 22:24:40.684139 1367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:40.713747 1367 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:40.714082 1367 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:40.718536 1367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54888c3e679f408e9e0b5618010cad2f: Generated new TSK 0
I20260811 22:24:40.719431 1367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:40.726382 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1199
2026-08-11T22:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1595 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:40 dist-test-slave-4tdj krb5kdc[1384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:40 dist-test-slave-4tdj krb5kdc[1384](info): set up 2 sockets
Aug 11 22:24:40 dist-test-slave-4tdj krb5kdc[1384](info): commencing operation
krb5kdc: starting...
W20260811 22:24:41.916961 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:42 dist-test-slave-4tdj krb5kdc[1384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:42.061074 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:42.179978 1414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:42.180292 1414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:42.180338 1414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:42.184571 1414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:42.184640 1414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:42.184659 1414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:42.184674 1414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:42.184695 1414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:42.190227 1414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:42.191375 1414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:42.192413 1414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:42.201319 1421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:42.201314 1420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:42.201309 1423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:42.201490 1414 server_base.cc:1034] running on GCE node
I20260811 22:24:42.202220 1414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:42.203462 1414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:42.204679 1414 hybrid_clock.cc:648] HybridClock initialized: now 1786487082204651 us; error 34 us; skew 500 ppm
Aug 11 22:24:42 dist-test-slave-4tdj krb5kdc[1384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:42.208189 1414 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:42.209190 1414 webserver.cc:460] Webserver started at http://127.0.0.1:37557/ using document root <none> and password file <none>
I20260811 22:24:42.209784 1414 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:42.209856 1414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:42.210073 1414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:42.212285 1414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "5d6b5e180a954b3a8eb596dea59ae2e8"
format_stamp: "Formatted at 2026-08-11 22:24:42 on dist-test-slave-4tdj"
server_key: "c8ea4c200bab620da9ab428f0221f1a7"
server_key_iv: "ba8323ebb8a6df38e6943a9e5c990b74"
server_key_version: "encryptionkey@0"
I20260811 22:24:42.212859 1414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "5d6b5e180a954b3a8eb596dea59ae2e8"
format_stamp: "Formatted at 2026-08-11 22:24:42 on dist-test-slave-4tdj"
server_key: "c8ea4c200bab620da9ab428f0221f1a7"
server_key_iv: "ba8323ebb8a6df38e6943a9e5c990b74"
server_key_version: "encryptionkey@0"
I20260811 22:24:42.216732 1414 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:42.219369 1433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:42.220625 1414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:24:42.220753 1414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "5d6b5e180a954b3a8eb596dea59ae2e8"
format_stamp: "Formatted at 2026-08-11 22:24:42 on dist-test-slave-4tdj"
server_key: "c8ea4c200bab620da9ab428f0221f1a7"
server_key_iv: "ba8323ebb8a6df38e6943a9e5c990b74"
server_key_version: "encryptionkey@0"
I20260811 22:24:42.220857 1414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:42.241211 1414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:42.244228 1414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:42.244424 1414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:42.251991 1414 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44995
I20260811 22:24:42.252007 1537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44995 every 8 connection(s)
I20260811 22:24:42.253013 1414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:42.256007 1538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:42.258584 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1414
I20260811 22:24:42.258728 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:42.258985 29709 external_mini_cluster.cc:1428] Setting key e2c0660a21814827838168a5280bdb8d
I20260811 22:24:42.262418 1538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8: Bootstrap starting.
I20260811 22:24:42.265143 1538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:42.266047 1538 log.cc:824] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:24:42 dist-test-slave-4tdj krb5kdc[1384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:24:42.268442 1538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8: No bootstrap required, opened a new log
I20260811 22:24:42.272799 1538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44995 } }
I20260811 22:24:42.273175 1538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:42.273249 1538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d6b5e180a954b3a8eb596dea59ae2e8, State: Initialized, Role: FOLLOWER
I20260811 22:24:42.273715 1538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44995 } }
I20260811 22:24:42.273854 1538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:42.273921 1538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:42.274025 1538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:42.274179 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:42.260494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42879 (local address 127.0.0.1:44995)
0811 22:24:42.260912 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:42.260928 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:42.260969 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:24:42.261642 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:42.261662 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:42.262033 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:42.262366 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:42.263317 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:42.264388 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:42.265558 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:42.266077 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:42.266246 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:42.269071 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:42.269077 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:42.269093 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:42.269123 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:42.270981 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:42.271547 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:42.271552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:42.271558 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:42.271702 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:42.271994 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:42.271996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:42.271997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:42.272452 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:42.272624 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:42.272900 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 22:24:42.273225 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":119,"threads_started":1}
I20260811 22:24:42.275314 1538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44995 } }
I20260811 22:24:42.275672 1538 leader_election.cc:302] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d6b5e180a954b3a8eb596dea59ae2e8; no voters:
I20260811 22:24:42.276012 1538 leader_election.cc:288] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:42.276125 1546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:42.276394 1546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [term 1 LEADER]: Becoming Leader. State: Replica: 5d6b5e180a954b3a8eb596dea59ae2e8, State: Running, Role: LEADER
I20260811 22:24:42.276793 1546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44995 } }
I20260811 22:24:42.277370 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:42.277431 1538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:42.278100 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d6b5e180a954b3a8eb596dea59ae2e8. Latest consensus state: current_term: 1 leader_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44995 } } }
I20260811 22:24:42.278256 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:42.278626 1547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d6b5e180a954b3a8eb596dea59ae2e8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44995 } } }
I20260811 22:24:42.278728 1547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:42.278852 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:42.282531 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:42.289819 1542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:42.279741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42597 (local address 127.0.0.1:44995)
0811 22:24:42.279857 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:42.279862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:42.280249 (+ 387us) server_negotiation.cc:407] Connection header received
0811 22:24:42.280369 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:42.280372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:42.280423 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:42.280495 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:42.281428 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:42.282225 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:42.283463 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:42.283805 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:42.283914 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:42.285585 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:42.285588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:42.285590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:42.285615 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:42.287908 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:42.288434 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:42.288438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:42.288440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:42.288534 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:42.288848 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:42.288851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:42.288853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:42.289114 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:42.289241 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:42.289520 (+ 279us) server_negotiation.cc:299] Negotiation successful
0811 22:24:42.289658 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:24:42.291604 1559 catalog_manager.cc:1347] Generated new cluster ID: a7f079dfbc5f4f7c89ed607bbcda0783
I20260811 22:24:42.291679 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:42.317188 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:42.318645 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:42.332820 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d6b5e180a954b3a8eb596dea59ae2e8: Generated new TSK 0
I20260811 22:24:42.333765 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:42.338418 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1414
2026-08-11T22:24:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:42 dist-test-slave-4tdj krb5kdc[1589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:42 dist-test-slave-4tdj krb5kdc[1589](info): set up 2 sockets
Aug 11 22:24:42 dist-test-slave-4tdj krb5kdc[1589](info): commencing operation
krb5kdc: starting...
W20260811 22:24:43.542711 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:43 dist-test-slave-4tdj krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:43.673728 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:43.790571 1617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:43.790876 1617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:43.790921 1617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:43.794994 1617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:43.795076 1617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:43.795096 1617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:43.795112 1617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:43.795126 1617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:43.800688 1617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:43.801857 1617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:43.802949 1617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:43.811985 1627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:43.812119 1617 server_base.cc:1034] running on GCE node
W20260811 22:24:43.812136 1635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:43.812335 1629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:43.812942 1617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:43.814133 1617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:43.815296 1617 hybrid_clock.cc:648] HybridClock initialized: now 1786487083815277 us; error 37 us; skew 500 ppm
Aug 11 22:24:43 dist-test-slave-4tdj krb5kdc[1589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:43.818068 1617 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:43.819077 1617 webserver.cc:460] Webserver started at http://127.0.0.1:45705/ using document root <none> and password file <none>
I20260811 22:24:43.819648 1617 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:43.819710 1617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:43.819918 1617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:43.822024 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "2311bab0a51445a483e27e89199640b6"
format_stamp: "Formatted at 2026-08-11 22:24:43 on dist-test-slave-4tdj"
server_key: "ddb87a706ef6499c37c0540f7ca7b184"
server_key_iv: "de275cb3c167c52ceed88cddcdc151c4"
server_key_version: "encryptionkey@0"
I20260811 22:24:43.822636 1617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "2311bab0a51445a483e27e89199640b6"
format_stamp: "Formatted at 2026-08-11 22:24:43 on dist-test-slave-4tdj"
server_key: "ddb87a706ef6499c37c0540f7ca7b184"
server_key_iv: "de275cb3c167c52ceed88cddcdc151c4"
server_key_version: "encryptionkey@0"
I20260811 22:24:43.826500 1617 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:24:43.828929 1647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:43.830215 1617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:24:43.830402 1617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "2311bab0a51445a483e27e89199640b6"
format_stamp: "Formatted at 2026-08-11 22:24:43 on dist-test-slave-4tdj"
server_key: "ddb87a706ef6499c37c0540f7ca7b184"
server_key_iv: "de275cb3c167c52ceed88cddcdc151c4"
server_key_version: "encryptionkey@0"
I20260811 22:24:43.830514 1617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:43.845593 1617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:43.848742 1617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:43.848919 1617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:43.856467 1617 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36425
I20260811 22:24:43.856477 1737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36425 every 8 connection(s)
I20260811 22:24:43.857517 1617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:43.860603 1739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:43.861444 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1617
I20260811 22:24:43.861620 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:43.861879 29709 external_mini_cluster.cc:1428] Setting key f792505a44dc63b61dea7e25568d9bae
I20260811 22:24:43.867240 1739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6: Bootstrap starting.
I20260811 22:24:43.870100 1739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:43.871021 1739 log.cc:824] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:43.871491 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:43.863368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53421 (local address 127.0.0.1:36425)
0811 22:24:43.863831 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:43.863847 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:43.863884 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:24:43.864531 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:43.864552 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:43.864929 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:43.865238 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:43.865642 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:43.866689 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:43.867896 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:43.868402 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:43.868527 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:43.869492 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:43.869498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:43.869513 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:43.869932 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:43.869994 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:43.870019 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:43.870122 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:43.870383 (+ 261us) server_negotiation.cc:299] Negotiation successful
0811 22:24:43.870654 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":132,"threads_started":1}
I20260811 22:24:43.873394 1739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6: No bootstrap required, opened a new log
I20260811 22:24:43.874707 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:43.877606 1739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2311bab0a51445a483e27e89199640b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36425 } }
I20260811 22:24:43.877883 1739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:43.877944 1739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2311bab0a51445a483e27e89199640b6, State: Initialized, Role: FOLLOWER
I20260811 22:24:43.878454 1739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2311bab0a51445a483e27e89199640b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36425 } }
I20260811 22:24:43.878613 1739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:43.878685 1739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:43.878789 1739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:43.879787 1739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2311bab0a51445a483e27e89199640b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36425 } }
I20260811 22:24:43.880158 1739 leader_election.cc:302] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2311bab0a51445a483e27e89199640b6; no voters:
I20260811 22:24:43.880539 1739 leader_election.cc:288] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:43.880604 1755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:43.880817 1742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:43.876822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47547 (local address 127.0.0.1:36425)
0811 22:24:43.876944 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:43.876949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:43.877128 (+ 179us) server_negotiation.cc:407] Connection header received
0811 22:24:43.877258 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:43.877261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:43.877307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:43.877374 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:43.877658 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:43.878263 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:43.879238 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:43.879498 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:43.879589 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:43.880390 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:43.880393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:43.880396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:43.880410 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:43.880437 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:43.880448 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:43.880509 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:43.880621 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 22:24:43.880678 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 22:24:43.881014 1755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2311bab0a51445a483e27e89199640b6, State: Running, Role: LEADER
I20260811 22:24:43.881364 1755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2311bab0a51445a483e27e89199640b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36425 } }
I20260811 22:24:43.882122 1739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:43.882794 1755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2311bab0a51445a483e27e89199640b6. Latest consensus state: current_term: 1 leader_uuid: "2311bab0a51445a483e27e89199640b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2311bab0a51445a483e27e89199640b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36425 } } }
I20260811 22:24:43.882949 1755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:43.882786 1756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2311bab0a51445a483e27e89199640b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2311bab0a51445a483e27e89199640b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36425 } } }
I20260811 22:24:43.883340 1756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:43.884631 1768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:43.887992 1768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:43.893780 1768 catalog_manager.cc:1347] Generated new cluster ID: 66586253b22f41cda7817225db38fb7a
I20260811 22:24:43.893869 1768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:43.903855 1768 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:43.904887 1768 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:43.912839 1768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2311bab0a51445a483e27e89199640b6: Generated new TSK 0
I20260811 22:24:43.913645 1768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:43.918747 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1617
2026-08-11T22:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1572 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:43 dist-test-slave-4tdj krb5kdc[1792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:43 dist-test-slave-4tdj krb5kdc[1792](info): set up 2 sockets
Aug 11 22:24:43 dist-test-slave-4tdj krb5kdc[1792](info): commencing operation
krb5kdc: starting...
W20260811 22:24:45.113333 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:45 dist-test-slave-4tdj krb5kdc[1792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:45.250029 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:45.368121 1811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:45.368425 1811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:45.368489 1811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:45.372648 1811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:45.372727 1811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:45.372756 1811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:45.372781 1811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:45.372810 1811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:45.378432 1811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:45.379601 1811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:45.380682 1811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:45.389514 1818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:45.389518 1821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:45.389514 1826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:45.389742 1811 server_base.cc:1034] running on GCE node
I20260811 22:24:45.390566 1811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:45.391705 1811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:45.392874 1811 hybrid_clock.cc:648] HybridClock initialized: now 1786487085392855 us; error 29 us; skew 500 ppm
Aug 11 22:24:45 dist-test-slave-4tdj krb5kdc[1792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:45.395706 1811 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:45.396661 1811 webserver.cc:460] Webserver started at http://127.0.0.1:36135/ using document root <none> and password file <none>
I20260811 22:24:45.397228 1811 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:45.397301 1811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:45.397519 1811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:45.399732 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "cec9552ee23b4fb0988bad6738cbb24c"
format_stamp: "Formatted at 2026-08-11 22:24:45 on dist-test-slave-4tdj"
server_key: "c8e220725ea2b56896d429c538efb18e"
server_key_iv: "c7dba3e140e264bfeeadcb2c68201aff"
server_key_version: "encryptionkey@0"
I20260811 22:24:45.400321 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "cec9552ee23b4fb0988bad6738cbb24c"
format_stamp: "Formatted at 2026-08-11 22:24:45 on dist-test-slave-4tdj"
server_key: "c8e220725ea2b56896d429c538efb18e"
server_key_iv: "c7dba3e140e264bfeeadcb2c68201aff"
server_key_version: "encryptionkey@0"
I20260811 22:24:45.404469 1811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:45.407141 1838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:45.408475 1811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:45.408629 1811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "cec9552ee23b4fb0988bad6738cbb24c"
format_stamp: "Formatted at 2026-08-11 22:24:45 on dist-test-slave-4tdj"
server_key: "c8e220725ea2b56896d429c538efb18e"
server_key_iv: "c7dba3e140e264bfeeadcb2c68201aff"
server_key_version: "encryptionkey@0"
I20260811 22:24:45.408743 1811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:45.413450 1811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:45.416363 1811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:45.416540 1811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:45.423834 1811 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42055
I20260811 22:24:45.423846 1934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42055 every 8 connection(s)
I20260811 22:24:45.424816 1811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:45.427712 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 1811
I20260811 22:24:45.427845 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:45.427838 1937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:45.428108 29709 external_mini_cluster.cc:1428] Setting key e2c80a5874889f42bcfe03ef12c59ba4
I20260811 22:24:45.433674 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:45.429544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37027 (local address 127.0.0.1:42055)
0811 22:24:45.429983 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:45.430002 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:45.430046 (+ 44us) server_negotiation.cc:407] Connection header received
0811 22:24:45.430759 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:45.430784 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:45.431169 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:45.431560 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:45.432232 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:45.432238 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:45.432254 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:45.432265 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:45.432323 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:45.432350 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:45.432488 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:45.432560 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 22:24:45.432729 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260811 22:24:45.435482 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c: Bootstrap starting.
I20260811 22:24:45.436700 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:45.437980 1937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:45.438823 1937 log.cc:824] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:45.440434 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:45.438810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39227 (local address 127.0.0.1:42055)
0811 22:24:45.438937 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:45.438942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:45.439120 (+ 178us) server_negotiation.cc:407] Connection header received
0811 22:24:45.439278 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:45.439280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:45.439314 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:45.439380 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:45.440086 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:45.440089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:45.440091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:45.440101 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:45.440126 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:45.440135 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:45.440206 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:45.440291 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 22:24:45.440322 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:24:45.441370 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c: No bootstrap required, opened a new log
I20260811 22:24:45.445075 1937 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9552ee23b4fb0988bad6738cbb24c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42055 } }
I20260811 22:24:45.445338 1937 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:45.445391 1937 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec9552ee23b4fb0988bad6738cbb24c, State: Initialized, Role: FOLLOWER
I20260811 22:24:45.445871 1937 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9552ee23b4fb0988bad6738cbb24c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42055 } }
I20260811 22:24:45.446039 1937 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:45.446108 1937 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:45.446221 1937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:45.447187 1937 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9552ee23b4fb0988bad6738cbb24c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42055 } }
I20260811 22:24:45.447537 1937 leader_election.cc:302] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec9552ee23b4fb0988bad6738cbb24c; no voters:
I20260811 22:24:45.447867 1937 leader_election.cc:288] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:45.447986 1963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:45.448235 1963 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [term 1 LEADER]: Becoming Leader. State: Replica: cec9552ee23b4fb0988bad6738cbb24c, State: Running, Role: LEADER
I20260811 22:24:45.448575 1963 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9552ee23b4fb0988bad6738cbb24c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42055 } }
I20260811 22:24:45.449136 1937 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:45.450218 1965 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec9552ee23b4fb0988bad6738cbb24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9552ee23b4fb0988bad6738cbb24c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42055 } } }
I20260811 22:24:45.450388 1965 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:45.450583 1966 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec9552ee23b4fb0988bad6738cbb24c. Latest consensus state: current_term: 1 leader_uuid: "cec9552ee23b4fb0988bad6738cbb24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9552ee23b4fb0988bad6738cbb24c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42055 } } }
I20260811 22:24:45.450736 1966 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:45.450817 1977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:45.454090 1977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:45.459908 1977 catalog_manager.cc:1347] Generated new cluster ID: 8b0623632c704c4bb1b71f29f12d5180
I20260811 22:24:45.459978 1977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:45.471745 1977 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:45.472078 1977 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:45.478260 1977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec9552ee23b4fb0988bad6738cbb24c: Generated new TSK 0
I20260811 22:24:45.479014 1977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:45.490063 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 1811
2026-08-11T22:24:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1577 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:45 dist-test-slave-4tdj krb5kdc[1991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:45 dist-test-slave-4tdj krb5kdc[1991](info): set up 2 sockets
Aug 11 22:24:45 dist-test-slave-4tdj krb5kdc[1991](info): commencing operation
krb5kdc: starting...
W20260811 22:24:46.694909 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:46 dist-test-slave-4tdj krb5kdc[1991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:46.821848 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:46.936890 2014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:46.937188 2014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:46.937250 2014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:46.941380 2014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:46.941464 2014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:46.941493 2014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:46.941514 2014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:46.941534 2014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:46.947420 2014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:46.948598 2014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:46.949666 2014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:46.958640 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:46.958652 2031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:46.958685 2026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:46.958710 2014 server_base.cc:1034] running on GCE node
I20260811 22:24:46.959411 2014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:46.960690 2014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:46.961879 2014 hybrid_clock.cc:648] HybridClock initialized: now 1786487086961858 us; error 36 us; skew 500 ppm
Aug 11 22:24:46 dist-test-slave-4tdj krb5kdc[1991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:46.970317 2014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:46.971436 2014 webserver.cc:460] Webserver started at http://127.0.0.1:35573/ using document root <none> and password file <none>
I20260811 22:24:46.972036 2014 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:46.972103 2014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:46.972329 2014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:46.974653 2014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "94d7f3590b6749238282707d42e33d43"
format_stamp: "Formatted at 2026-08-11 22:24:46 on dist-test-slave-4tdj"
server_key: "f15eac3cf3d76ca0f05ff34c048ab7b2"
server_key_iv: "176c8e195775d428c26b59b335097c0a"
server_key_version: "encryptionkey@0"
I20260811 22:24:46.975246 2014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "94d7f3590b6749238282707d42e33d43"
format_stamp: "Formatted at 2026-08-11 22:24:46 on dist-test-slave-4tdj"
server_key: "f15eac3cf3d76ca0f05ff34c048ab7b2"
server_key_iv: "176c8e195775d428c26b59b335097c0a"
server_key_version: "encryptionkey@0"
I20260811 22:24:46.979331 2014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:24:46.981945 2048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:46.983351 2014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:46.983489 2014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "94d7f3590b6749238282707d42e33d43"
format_stamp: "Formatted at 2026-08-11 22:24:46 on dist-test-slave-4tdj"
server_key: "f15eac3cf3d76ca0f05ff34c048ab7b2"
server_key_iv: "176c8e195775d428c26b59b335097c0a"
server_key_version: "encryptionkey@0"
I20260811 22:24:46.983636 2014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:47.005401 2014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:47.008932 2014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:47.009150 2014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:47.016664 2014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41587
I20260811 22:24:47.016674 2134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41587 every 8 connection(s)
I20260811 22:24:47.017659 2014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:47.019145 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2014
I20260811 22:24:47.019272 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:47.019563 29709 external_mini_cluster.cc:1428] Setting key db748616d9fd468ada75d9662ea09d98
I20260811 22:24:47.021246 2137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:47.029186 2137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43: Bootstrap starting.
I20260811 22:24:47.031689 2137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43: Neither blocks nor log segments found. Creating new log.
Aug 11 22:24:47 dist-test-slave-4tdj krb5kdc[1991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:24:47.032493 2137 log.cc:824] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:47.034533 2137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43: No bootstrap required, opened a new log
I20260811 22:24:47.037592 2140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:47.021390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:51481 (local address 127.0.0.1:41587)
0811 22:24:47.023516 (+ 2126us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:47.023537 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:47.023585 (+ 48us) server_negotiation.cc:407] Connection header received
0811 22:24:47.024804 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:47.024833 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:47.025362 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:47.025776 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:47.027341 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:47.028707 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:47.029758 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:47.030273 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:47.030451 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:47.032942 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:47.032947 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:47.032962 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:47.032993 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:47.034712 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:47.035274 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:47.035279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:47.035285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:47.035394 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:47.035723 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:47.035725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:47.035726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:47.036113 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:47.036284 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:47.036573 (+ 289us) server_negotiation.cc:299] Negotiation successful
0811 22:24:47.036794 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1946,"thread_start_us":147,"threads_started":1}
I20260811 22:24:47.038672 2137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7f3590b6749238282707d42e33d43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41587 } }
I20260811 22:24:47.038913 2137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:47.038971 2137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d7f3590b6749238282707d42e33d43, State: Initialized, Role: FOLLOWER
I20260811 22:24:47.039417 2137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7f3590b6749238282707d42e33d43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41587 } }
I20260811 22:24:47.039561 2137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:47.039635 2137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:47.039741 2137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:47.040632 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:47.040860 2137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7f3590b6749238282707d42e33d43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41587 } }
I20260811 22:24:47.041204 2137 leader_election.cc:302] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d7f3590b6749238282707d42e33d43; no voters:
I20260811 22:24:47.041517 2137 leader_election.cc:288] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:47.041667 2143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:47.041873 2143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [term 1 LEADER]: Becoming Leader. State: Replica: 94d7f3590b6749238282707d42e33d43, State: Running, Role: LEADER
I20260811 22:24:47.042237 2143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7f3590b6749238282707d42e33d43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41587 } }
I20260811 22:24:47.042809 2137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:47.044011 2144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d7f3590b6749238282707d42e33d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7f3590b6749238282707d42e33d43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41587 } } }
I20260811 22:24:47.044154 2144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:47.044516 2150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d7f3590b6749238282707d42e33d43. Latest consensus state: current_term: 1 leader_uuid: "94d7f3590b6749238282707d42e33d43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d7f3590b6749238282707d42e33d43" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41587 } } }
I20260811 22:24:47.044611 2150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43 [sys.catalog]: This master's current role is: LEADER
W20260811 22:24:47.048231 2189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:24:47.048305 2189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:24:47.048386 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:47.049300 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:47.052362 2140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:47.042967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:56755 (local address 127.0.0.1:41587)
0811 22:24:47.043092 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:47.043097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:47.043516 (+ 419us) server_negotiation.cc:407] Connection header received
0811 22:24:47.043719 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:47.043724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:47.043778 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:47.043830 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:47.045022 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:47.045738 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:47.046653 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:47.046977 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:47.047079 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:47.048723 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:47.048727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:47.048730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:47.048756 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:47.050448 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:47.050999 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:47.051003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:47.051005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:47.051098 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:47.051439 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:47.051443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:47.051445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:47.051667 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:47.051822 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:47.052107 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 22:24:47.052211 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 22:24:47.055557 2179 catalog_manager.cc:1347] Generated new cluster ID: af0e532e3c214873ab03e54422a0a5aa
I20260811 22:24:47.055619 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:47.067308 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:47.068290 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:47.079744 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d7f3590b6749238282707d42e33d43: Generated new TSK 0
I20260811 22:24:47.080803 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:47.092785 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2014
W20260811 22:24:47.118502 1989 connection.cc:535] client connection to 127.0.0.1:41587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41587): Connection reset by peer (error 104)
2026-08-11T22:24:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:47 dist-test-slave-4tdj krb5kdc[2196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:47 dist-test-slave-4tdj krb5kdc[2196](info): set up 2 sockets
Aug 11 22:24:47 dist-test-slave-4tdj krb5kdc[2196](info): commencing operation
krb5kdc: starting...
W20260811 22:24:48.345018 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:48 dist-test-slave-4tdj krb5kdc[2196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:48.474627 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34317
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:48.589956 2222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:48.590261 2222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:48.590341 2222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:48.594439 2222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:48.594516 2222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:48.594542 2222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:48.594564 2222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:48.594594 2222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:48.600108 2222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:48.601274 2222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:48.602377 2222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:48.611287 2237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:48.611388 2222 server_base.cc:1034] running on GCE node
W20260811 22:24:48.611258 2241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:48.611265 2236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:48.612178 2222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:48.613288 2222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:48.614437 2222 hybrid_clock.cc:648] HybridClock initialized: now 1786487088614417 us; error 38 us; skew 500 ppm
Aug 11 22:24:48 dist-test-slave-4tdj krb5kdc[2196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:48.617156 2222 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:48.618112 2222 webserver.cc:460] Webserver started at http://127.0.0.1:38831/ using document root <none> and password file <none>
I20260811 22:24:48.618716 2222 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:48.618778 2222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:48.618954 2222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:48.621058 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "9d5357ed22be439c91c624fc13b2bf3a"
format_stamp: "Formatted at 2026-08-11 22:24:48 on dist-test-slave-4tdj"
server_key: "ef0f7e2fed34b1c4ed31b3361045f549"
server_key_iv: "cc1745ad760e60bbf9e6fae99b91e4c1"
server_key_version: "encryptionkey@0"
I20260811 22:24:48.621600 2222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "9d5357ed22be439c91c624fc13b2bf3a"
format_stamp: "Formatted at 2026-08-11 22:24:48 on dist-test-slave-4tdj"
server_key: "ef0f7e2fed34b1c4ed31b3361045f549"
server_key_iv: "cc1745ad760e60bbf9e6fae99b91e4c1"
server_key_version: "encryptionkey@0"
I20260811 22:24:48.625476 2222 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:24:48.628131 2248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:48.629436 2222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:24:48.629558 2222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "9d5357ed22be439c91c624fc13b2bf3a"
format_stamp: "Formatted at 2026-08-11 22:24:48 on dist-test-slave-4tdj"
server_key: "ef0f7e2fed34b1c4ed31b3361045f549"
server_key_iv: "cc1745ad760e60bbf9e6fae99b91e4c1"
server_key_version: "encryptionkey@0"
I20260811 22:24:48.629663 2222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:48.646883 2222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:48.650098 2222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:48.650328 2222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:48.657655 2222 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34317
I20260811 22:24:48.657673 2352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34317 every 8 connection(s)
I20260811 22:24:48.658669 2222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:48.661577 2353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:48.661931 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2222
I20260811 22:24:48.662051 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:48.662346 29709 external_mini_cluster.cc:1428] Setting key c5255405c71e9beec71b991c3a6fdf63
I20260811 22:24:48.668341 2353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a: Bootstrap starting.
I20260811 22:24:48.670779 2353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:48.671489 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:48.663778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:57095 (local address 127.0.0.1:34317)
0811 22:24:48.664271 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:48.664291 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:48.664332 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:24:48.665027 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:48.665046 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:48.665417 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:48.665753 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:48.666158 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:48.667186 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:48.668060 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:48.668560 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:48.668725 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:48.669635 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:48.669640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:48.669653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:48.670042 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:48.670098 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:48.670122 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:48.670233 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:48.670600 (+ 367us) server_negotiation.cc:299] Negotiation successful
0811 22:24:48.670736 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260811 22:24:48.671564 2353 log.cc:824] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:48.673669 2353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a: No bootstrap required, opened a new log
I20260811 22:24:48.674522 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:48.677682 2353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5357ed22be439c91c624fc13b2bf3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34317 } }
I20260811 22:24:48.677981 2353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:48.678057 2353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5357ed22be439c91c624fc13b2bf3a, State: Initialized, Role: FOLLOWER
I20260811 22:24:48.678534 2353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5357ed22be439c91c624fc13b2bf3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34317 } }
I20260811 22:24:48.678675 2353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:48.678745 2353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:48.678844 2353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:48.680039 2353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5357ed22be439c91c624fc13b2bf3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34317 } }
I20260811 22:24:48.680446 2353 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d5357ed22be439c91c624fc13b2bf3a; no voters:
I20260811 22:24:48.680611 2357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:48.676493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:50713 (local address 127.0.0.1:34317)
0811 22:24:48.676622 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:48.676626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:48.676726 (+ 100us) server_negotiation.cc:407] Connection header received
0811 22:24:48.676830 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:48.676833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:48.676883 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:48.676964 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:48.677164 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:48.677905 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:48.678917 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:48.679251 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:48.679356 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:48.680137 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:48.680141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:48.680144 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:48.680157 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:48.680186 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:48.680199 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:48.680283 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:48.680404 (+ 121us) server_negotiation.cc:299] Negotiation successful
0811 22:24:48.680441 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:24:48.680822 2353 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:48.680923 2366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:48.681140 2366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [term 1 LEADER]: Becoming Leader. State: Replica: 9d5357ed22be439c91c624fc13b2bf3a, State: Running, Role: LEADER
I20260811 22:24:48.681531 2366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5357ed22be439c91c624fc13b2bf3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34317 } }
I20260811 22:24:48.682129 2353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:48.682780 2366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d5357ed22be439c91c624fc13b2bf3a. Latest consensus state: current_term: 1 leader_uuid: "9d5357ed22be439c91c624fc13b2bf3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5357ed22be439c91c624fc13b2bf3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34317 } } }
I20260811 22:24:48.682952 2366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:48.683285 2367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d5357ed22be439c91c624fc13b2bf3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5357ed22be439c91c624fc13b2bf3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34317 } } }
I20260811 22:24:48.683458 2367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:48.683393 2381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:48.687487 2381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:48.693814 2381 catalog_manager.cc:1347] Generated new cluster ID: dcaedc9a1ad44297b9159d31c0c7a117
I20260811 22:24:48.693912 2381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:48.711103 2381 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:48.712106 2381 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:48.722839 2381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d5357ed22be439c91c624fc13b2bf3a: Generated new TSK 0
I20260811 22:24:48.723675 2381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:48.729768 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2222
W20260811 22:24:48.743413 2193 connection.cc:535] client connection to 127.0.0.1:34317 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34317): Connection reset by peer (error 104)
2026-08-11T22:24:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1624 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:48 dist-test-slave-4tdj krb5kdc[2410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:48 dist-test-slave-4tdj krb5kdc[2410](info): set up 2 sockets
Aug 11 22:24:48 dist-test-slave-4tdj krb5kdc[2410](info): commencing operation
krb5kdc: starting...
W20260811 22:24:49.921861 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.164s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:50 dist-test-slave-4tdj krb5kdc[2410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:50.053570 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33505
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:50.169919 2434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:50.170238 2434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:50.170331 2434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:50.174432 2434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:50.174504 2434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:50.174531 2434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:50.174553 2434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:50.174573 2434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:50.179989 2434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:50.181133 2434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:50.182184 2434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:50.190886 2442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:50.190934 2434 server_base.cc:1034] running on GCE node
W20260811 22:24:50.190865 2443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:50.190853 2446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:50.191725 2434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:50.192870 2434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:50.194059 2434 hybrid_clock.cc:648] HybridClock initialized: now 1786487090194044 us; error 28 us; skew 500 ppm
Aug 11 22:24:50 dist-test-slave-4tdj krb5kdc[2410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:50.196967 2434 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:50.197904 2434 webserver.cc:460] Webserver started at http://127.0.0.1:40379/ using document root <none> and password file <none>
I20260811 22:24:50.198503 2434 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:50.198576 2434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:50.198832 2434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:50.200922 2434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "124f8a5a359f4d228110c1681d7986d0"
format_stamp: "Formatted at 2026-08-11 22:24:50 on dist-test-slave-4tdj"
server_key: "94b6a5e2f6e38e8dfafccb0d542bc5b6"
server_key_iv: "25b35ba14712317daa4abe0663df4a0e"
server_key_version: "encryptionkey@0"
I20260811 22:24:50.201488 2434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "124f8a5a359f4d228110c1681d7986d0"
format_stamp: "Formatted at 2026-08-11 22:24:50 on dist-test-slave-4tdj"
server_key: "94b6a5e2f6e38e8dfafccb0d542bc5b6"
server_key_iv: "25b35ba14712317daa4abe0663df4a0e"
server_key_version: "encryptionkey@0"
I20260811 22:24:50.205367 2434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260811 22:24:50.207980 2467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:50.209307 2434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:24:50.209431 2434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "124f8a5a359f4d228110c1681d7986d0"
format_stamp: "Formatted at 2026-08-11 22:24:50 on dist-test-slave-4tdj"
server_key: "94b6a5e2f6e38e8dfafccb0d542bc5b6"
server_key_iv: "25b35ba14712317daa4abe0663df4a0e"
server_key_version: "encryptionkey@0"
I20260811 22:24:50.209537 2434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:50.214159 2434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:50.217317 2434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:50.217494 2434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:50.224704 2434 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33505
I20260811 22:24:50.224725 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33505 every 8 connection(s)
I20260811 22:24:50.225732 2434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:50.228724 2565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:50.231166 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2434
I20260811 22:24:50.231312 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:50.231580 29709 external_mini_cluster.cc:1428] Setting key be9c8fc8dcc9a4a7d0d6e1277e01ef9c
I20260811 22:24:50.234778 2565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0: Bootstrap starting.
I20260811 22:24:50.237002 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:50.233018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:44249 (local address 127.0.0.1:33505)
0811 22:24:50.233466 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:50.233493 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:50.233532 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:24:50.234184 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:50.234203 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:50.234562 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:50.234866 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:50.235728 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:50.235733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:50.235746 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:50.235757 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:50.235816 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:50.235839 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:50.235960 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:50.236071 (+ 111us) server_negotiation.cc:299] Negotiation successful
0811 22:24:50.236236 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260811 22:24:50.237363 2565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:50.238188 2565 log.cc:824] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:50.240052 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:50.240420 2565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0: No bootstrap required, opened a new log
I20260811 22:24:50.243795 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:50.242088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:43633 (local address 127.0.0.1:33505)
0811 22:24:50.242216 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:50.242220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:50.242416 (+ 196us) server_negotiation.cc:407] Connection header received
0811 22:24:50.242545 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:50.242547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:50.242579 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:50.242642 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:50.243399 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:50.243403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:50.243405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:50.243417 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:50.243451 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:50.243463 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:50.243555 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:50.243632 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 22:24:50.243675 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:24:50.244638 2565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124f8a5a359f4d228110c1681d7986d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260811 22:24:50.244906 2565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:50.244964 2565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124f8a5a359f4d228110c1681d7986d0, State: Initialized, Role: FOLLOWER
I20260811 22:24:50.245417 2565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124f8a5a359f4d228110c1681d7986d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260811 22:24:50.245551 2565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:50.245618 2565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:50.245720 2565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:50.246765 2565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124f8a5a359f4d228110c1681d7986d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260811 22:24:50.247107 2565 leader_election.cc:302] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 124f8a5a359f4d228110c1681d7986d0; no voters:
I20260811 22:24:50.247418 2565 leader_election.cc:288] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:50.247527 2587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:50.247756 2587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [term 1 LEADER]: Becoming Leader. State: Replica: 124f8a5a359f4d228110c1681d7986d0, State: Running, Role: LEADER
I20260811 22:24:50.248155 2587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124f8a5a359f4d228110c1681d7986d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } }
I20260811 22:24:50.248679 2565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:50.250097 2588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "124f8a5a359f4d228110c1681d7986d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124f8a5a359f4d228110c1681d7986d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } } }
I20260811 22:24:50.250097 2590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124f8a5a359f4d228110c1681d7986d0. Latest consensus state: current_term: 1 leader_uuid: "124f8a5a359f4d228110c1681d7986d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "124f8a5a359f4d228110c1681d7986d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33505 } } }
I20260811 22:24:50.250252 2590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:50.250252 2588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:50.250638 2614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:50.253849 2614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:50.259662 2614 catalog_manager.cc:1347] Generated new cluster ID: 27f7f7d76b7f49ebabe5614ca6192bc1
I20260811 22:24:50.259755 2614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:50.278604 2614 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:50.278981 2614 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:50.284670 2614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 124f8a5a359f4d228110c1681d7986d0: Generated new TSK 0
I20260811 22:24:50.285624 2614 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:50.298676 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2434
2026-08-11T22:24:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1572 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:50 dist-test-slave-4tdj krb5kdc[2641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:50 dist-test-slave-4tdj krb5kdc[2641](info): set up 2 sockets
Aug 11 22:24:50 dist-test-slave-4tdj krb5kdc[2641](info): commencing operation
krb5kdc: starting...
W20260811 22:24:51.495142 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:51 dist-test-slave-4tdj krb5kdc[2641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:51.625289 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:51.742609 2666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:51.742913 2666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:51.742955 2666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:51.747028 2666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:51.747099 2666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:51.747119 2666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:51.747134 2666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:51.747148 2666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:51.752626 2666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:51.753739 2666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:51.754768 2666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:51.763481 2675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:51.763466 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:51.763540 2666 server_base.cc:1034] running on GCE node
W20260811 22:24:51.763482 2681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:51.764401 2666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:51.765648 2666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:51.766825 2666 hybrid_clock.cc:648] HybridClock initialized: now 1786487091766807 us; error 30 us; skew 500 ppm
Aug 11 22:24:51 dist-test-slave-4tdj krb5kdc[2641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:51.769731 2666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:51.770845 2666 webserver.cc:460] Webserver started at http://127.0.0.1:38399/ using document root <none> and password file <none>
I20260811 22:24:51.771440 2666 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:51.771502 2666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:51.771700 2666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:51.773895 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "a6416713cd50445ea279127a58b3c0f0"
format_stamp: "Formatted at 2026-08-11 22:24:51 on dist-test-slave-4tdj"
server_key: "5159e68a14c65954f44f4eed75fc2a18"
server_key_iv: "12fc6d106db52d755dbb1ba3753ceefa"
server_key_version: "encryptionkey@0"
I20260811 22:24:51.774505 2666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "a6416713cd50445ea279127a58b3c0f0"
format_stamp: "Formatted at 2026-08-11 22:24:51 on dist-test-slave-4tdj"
server_key: "5159e68a14c65954f44f4eed75fc2a18"
server_key_iv: "12fc6d106db52d755dbb1ba3753ceefa"
server_key_version: "encryptionkey@0"
I20260811 22:24:51.778355 2666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:24:51.780891 2698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:51.782375 2666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:51.782502 2666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "a6416713cd50445ea279127a58b3c0f0"
format_stamp: "Formatted at 2026-08-11 22:24:51 on dist-test-slave-4tdj"
server_key: "5159e68a14c65954f44f4eed75fc2a18"
server_key_iv: "12fc6d106db52d755dbb1ba3753ceefa"
server_key_version: "encryptionkey@0"
I20260811 22:24:51.782617 2666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:51.793351 2666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:51.796725 2666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:51.796914 2666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:51.804172 2666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40795
I20260811 22:24:51.804211 2795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40795 every 8 connection(s)
I20260811 22:24:51.805217 2666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:51.808336 2800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:51.812862 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2666
I20260811 22:24:51.813002 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:51.813295 29709 external_mini_cluster.cc:1428] Setting key 7b73cca03eec737ede6564c75fd60032
I20260811 22:24:51.814759 2800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0: Bootstrap starting.
I20260811 22:24:51.817226 2800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:51.817986 2800 log.cc:824] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:51.820025 2800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0: No bootstrap required, opened a new log
Aug 11 22:24:51 dist-test-slave-4tdj krb5kdc[2641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 22:24:51.824031 2800 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6416713cd50445ea279127a58b3c0f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260811 22:24:51.824411 2800 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:51.824493 2800 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6416713cd50445ea279127a58b3c0f0, State: Initialized, Role: FOLLOWER
I20260811 22:24:51.825014 2800 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6416713cd50445ea279127a58b3c0f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260811 22:24:51.825145 2800 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:51.825207 2800 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:51.825325 2800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:51.826261 2800 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6416713cd50445ea279127a58b3c0f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260811 22:24:51.826650 2800 leader_election.cc:302] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6416713cd50445ea279127a58b3c0f0; no voters:
I20260811 22:24:51.826992 2800 leader_election.cc:288] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:51.827113 2808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:51.828473 2800 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:51.828661 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:51.814927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:34931 (local address 127.0.0.1:40795)
0811 22:24:51.815387 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:51.815403 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:51.815442 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:24:51.816141 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:51.816160 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:51.816540 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:51.816866 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:51.817728 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:51.818978 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:51.819809 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:51.820308 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:51.820468 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:51.823268 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:51.823274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:51.823290 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:51.823321 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:51.825378 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:51.825901 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:51.825907 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:51.825914 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:51.826040 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:51.826387 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:51.826391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:51.826392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:51.826853 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:51.827068 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:51.827444 (+ 376us) server_negotiation.cc:299] Negotiation successful
0811 22:24:51.827684 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":135,"threads_started":1}
I20260811 22:24:51.830241 2808 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [term 1 LEADER]: Becoming Leader. State: Replica: a6416713cd50445ea279127a58b3c0f0, State: Running, Role: LEADER
I20260811 22:24:51.830647 2808 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6416713cd50445ea279127a58b3c0f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } }
I20260811 22:24:51.832130 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:51.832310 2822 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6416713cd50445ea279127a58b3c0f0. Latest consensus state: current_term: 1 leader_uuid: "a6416713cd50445ea279127a58b3c0f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6416713cd50445ea279127a58b3c0f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } } }
I20260811 22:24:51.832435 2822 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:51.832758 2809 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6416713cd50445ea279127a58b3c0f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6416713cd50445ea279127a58b3c0f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40795 } } }
I20260811 22:24:51.832809 2834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:51.832865 2809 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:51.836171 2834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:51.842146 2834 catalog_manager.cc:1347] Generated new cluster ID: 6b97aa05a5044e43bc3875b9935082f8
I20260811 22:24:51.842226 2834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:51.843007 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:51.834487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:36817 (local address 127.0.0.1:40795)
0811 22:24:51.834597 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:51.834601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:51.834810 (+ 209us) server_negotiation.cc:407] Connection header received
0811 22:24:51.834922 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:51.834925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:51.834971 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:51.835037 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:51.835697 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:51.836286 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:51.837094 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:51.837339 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:51.837425 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:51.839094 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:51.839098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:51.839101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:51.839128 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:51.841163 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:51.841643 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:51.841646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:51.841648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:51.841735 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:51.842107 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:51.842110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:51.842111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:51.842367 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:51.842503 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:51.842762 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 22:24:51.842860 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 22:24:51.874049 2834 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:51.875126 2834 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:51.881134 2834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6416713cd50445ea279127a58b3c0f0: Generated new TSK 0
I20260811 22:24:51.881896 2834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:51.889719 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2666
W20260811 22:24:51.900149 2639 connection.cc:535] client connection to 127.0.0.1:40795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40795): Connection reset by peer (error 104)
2026-08-11T22:24:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1584 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:51 dist-test-slave-4tdj krb5kdc[2859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:51 dist-test-slave-4tdj krb5kdc[2859](info): set up 2 sockets
Aug 11 22:24:51 dist-test-slave-4tdj krb5kdc[2859](info): commencing operation
krb5kdc: starting...
W20260811 22:24:53.106422 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:53 dist-test-slave-4tdj krb5kdc[2859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:53.238163 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44513
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:53.359624 2888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:53.359961 2888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:53.360023 2888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:53.364246 2888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:53.364338 2888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:53.364365 2888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:53.364394 2888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:53.364423 2888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:53.370071 2888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44513
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:53.371346 2888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:53.372516 2888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:53.381554 2898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:53.381659 2888 server_base.cc:1034] running on GCE node
W20260811 22:24:53.381563 2894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:53.381577 2895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:53.382418 2888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:53.383606 2888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:53.384783 2888 hybrid_clock.cc:648] HybridClock initialized: now 1786487093384767 us; error 28 us; skew 500 ppm
Aug 11 22:24:53 dist-test-slave-4tdj krb5kdc[2859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:53.390920 2888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:53.391950 2888 webserver.cc:460] Webserver started at http://127.0.0.1:38183/ using document root <none> and password file <none>
I20260811 22:24:53.392513 2888 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:53.392573 2888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:53.392767 2888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:53.394928 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "0fcf9d8aca1e4c439e46f543e709e514"
format_stamp: "Formatted at 2026-08-11 22:24:53 on dist-test-slave-4tdj"
server_key: "d7cbdff898887aebf0b45c27db10602b"
server_key_iv: "159d7f6ef3914bffa4d75db34e6f2b5e"
server_key_version: "encryptionkey@0"
I20260811 22:24:53.395484 2888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "0fcf9d8aca1e4c439e46f543e709e514"
format_stamp: "Formatted at 2026-08-11 22:24:53 on dist-test-slave-4tdj"
server_key: "d7cbdff898887aebf0b45c27db10602b"
server_key_iv: "159d7f6ef3914bffa4d75db34e6f2b5e"
server_key_version: "encryptionkey@0"
I20260811 22:24:53.399323 2888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:24:53.401784 2909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:53.403090 2888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 22:24:53.403206 2888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "0fcf9d8aca1e4c439e46f543e709e514"
format_stamp: "Formatted at 2026-08-11 22:24:53 on dist-test-slave-4tdj"
server_key: "d7cbdff898887aebf0b45c27db10602b"
server_key_iv: "159d7f6ef3914bffa4d75db34e6f2b5e"
server_key_version: "encryptionkey@0"
I20260811 22:24:53.403316 2888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:53.423669 2888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:53.427114 2888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:53.427322 2888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:53.434670 2888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44513
I20260811 22:24:53.434732 2997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44513 every 8 connection(s)
I20260811 22:24:53.435688 2888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:53.436246 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 2888
I20260811 22:24:53.436383 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:53.436655 29709 external_mini_cluster.cc:1428] Setting key fde1f5d2b2a250c1da9e760df13a4a01
I20260811 22:24:53.438951 2998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:53.444795 2998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514: Bootstrap starting.
I20260811 22:24:53.445698 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:53.438097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:42043 (local address 127.0.0.1:44513)
0811 22:24:53.438593 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:53.438609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:53.438646 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:24:53.439345 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:53.439364 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:53.439726 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:53.440038 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:53.440441 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:53.441429 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:53.442325 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:53.442807 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:53.442969 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:53.443937 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:53.443942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:53.443956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:53.444363 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:53.444421 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:53.444445 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:53.444565 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:53.444759 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 22:24:53.444904 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":183,"threads_started":1}
I20260811 22:24:53.447469 2998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:53.448342 2998 log.cc:824] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:53.448945 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:53.450759 2998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514: No bootstrap required, opened a new log
I20260811 22:24:53.455642 3002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:53.451280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:45179 (local address 127.0.0.1:44513)
0811 22:24:53.451402 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:53.451407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:53.451554 (+ 147us) server_negotiation.cc:407] Connection header received
0811 22:24:53.451653 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:53.451656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:53.451712 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:53.451796 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:53.452020 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:53.452800 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:53.453679 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:53.454028 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:53.454138 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:53.455044 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:53.455048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:53.455051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:53.455067 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:53.455102 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:53.455117 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:53.455227 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:53.455417 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 22:24:53.455489 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:24:53.455400 2998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcf9d8aca1e4c439e46f543e709e514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260811 22:24:53.455726 2998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:53.455799 2998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcf9d8aca1e4c439e46f543e709e514, State: Initialized, Role: FOLLOWER
I20260811 22:24:53.456293 2998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcf9d8aca1e4c439e46f543e709e514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260811 22:24:53.456431 2998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:53.456491 2998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:53.456594 2998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:53.457572 2998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcf9d8aca1e4c439e46f543e709e514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260811 22:24:53.457922 2998 leader_election.cc:302] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fcf9d8aca1e4c439e46f543e709e514; no voters:
I20260811 22:24:53.458350 2998 leader_election.cc:288] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:53.458403 3016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:53.458639 3016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [term 1 LEADER]: Becoming Leader. State: Replica: 0fcf9d8aca1e4c439e46f543e709e514, State: Running, Role: LEADER
I20260811 22:24:53.458961 3016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcf9d8aca1e4c439e46f543e709e514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } }
I20260811 22:24:53.459743 2998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:53.460994 3018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fcf9d8aca1e4c439e46f543e709e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcf9d8aca1e4c439e46f543e709e514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } } }
I20260811 22:24:53.461129 3018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:53.460975 3017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fcf9d8aca1e4c439e46f543e709e514. Latest consensus state: current_term: 1 leader_uuid: "0fcf9d8aca1e4c439e46f543e709e514" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fcf9d8aca1e4c439e46f543e709e514" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44513 } } }
I20260811 22:24:53.461265 3017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:53.461566 3027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:53.464473 3027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:53.470916 3027 catalog_manager.cc:1347] Generated new cluster ID: 77c55395b04749ae8cac6b37da2228a5
I20260811 22:24:53.470992 3027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:53.478747 3027 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:53.480037 3027 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:53.498747 3027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fcf9d8aca1e4c439e46f543e709e514: Generated new TSK 0
I20260811 22:24:53.499712 3027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:53.507143 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 2888
W20260811 22:24:53.518836 2854 connection.cc:535] client connection to 127.0.0.1:44513 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44513): Connection reset by peer (error 104)
2026-08-11T22:24:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1618 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:53 dist-test-slave-4tdj krb5kdc[3049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:53 dist-test-slave-4tdj krb5kdc[3049](info): set up 2 sockets
Aug 11 22:24:53 dist-test-slave-4tdj krb5kdc[3049](info): commencing operation
krb5kdc: starting...
W20260811 22:24:54.787182 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.254s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:54 dist-test-slave-4tdj krb5kdc[3049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 22:24:54.921880 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43617
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:55.042871 3075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:55.043193 3075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:55.043257 3075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:55.047508 3075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:55.047607 3075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:55.047636 3075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:55.047662 3075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:55.047693 3075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:55.053400 3075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:55.054630 3075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:55.055712 3075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:55.064869 3087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:55.064980 3075 server_base.cc:1034] running on GCE node
W20260811 22:24:55.064868 3090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:55.064874 3084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:55.065853 3075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:55.067212 3075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:55.068392 3075 hybrid_clock.cc:648] HybridClock initialized: now 1786487095068376 us; error 34 us; skew 500 ppm
Aug 11 22:24:55 dist-test-slave-4tdj krb5kdc[3049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:55.071409 3075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 22:24:55.072389 3075 webserver.cc:460] Webserver started at http://127.0.0.1:44877/ using document root <none> and password file <none>
I20260811 22:24:55.072945 3075 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:55.073006 3075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:55.073190 3075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:55.075448 3075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "b8e66805e45c4514981560743a8d458f"
format_stamp: "Formatted at 2026-08-11 22:24:55 on dist-test-slave-4tdj"
server_key: "2d05692c5bbf2baac425c91e028e4bc7"
server_key_iv: "81202d2be498aaac42324f1e780c75b5"
server_key_version: "encryptionkey@0"
I20260811 22:24:55.075994 3075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "b8e66805e45c4514981560743a8d458f"
format_stamp: "Formatted at 2026-08-11 22:24:55 on dist-test-slave-4tdj"
server_key: "2d05692c5bbf2baac425c91e028e4bc7"
server_key_iv: "81202d2be498aaac42324f1e780c75b5"
server_key_version: "encryptionkey@0"
I20260811 22:24:55.080036 3075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:55.082813 3104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:55.084251 3075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:55.084383 3075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "b8e66805e45c4514981560743a8d458f"
format_stamp: "Formatted at 2026-08-11 22:24:55 on dist-test-slave-4tdj"
server_key: "2d05692c5bbf2baac425c91e028e4bc7"
server_key_iv: "81202d2be498aaac42324f1e780c75b5"
server_key_version: "encryptionkey@0"
I20260811 22:24:55.084497 3075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:55.089547 3075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:55.092945 3075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:55.093135 3075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:55.100764 3075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43617
I20260811 22:24:55.100914 3210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43617 every 8 connection(s)
I20260811 22:24:55.101857 3075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:55.104969 3211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:55.109841 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3075
I20260811 22:24:55.109963 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:55.110232 29709 external_mini_cluster.cc:1428] Setting key 072f430671950180ee0fe33428a461ed
I20260811 22:24:55.111248 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f: Bootstrap starting.
I20260811 22:24:55.113720 3211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:55.114737 3211 log.cc:824] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:55.115525 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:55.111660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:54889 (local address 127.0.0.1:43617)
0811 22:24:55.112083 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:55.112099 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:55.112142 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:24:55.112809 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:55.112829 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:55.113119 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:55.113417 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:55.114250 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:55.114254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:55.114267 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:55.114276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:55.114360 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:55.114384 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:55.114468 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:55.114598 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 22:24:55.114762 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":106,"threads_started":1}
I20260811 22:24:55.116947 3211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f: No bootstrap required, opened a new log
I20260811 22:24:55.118330 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:55.121052 3211 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e66805e45c4514981560743a8d458f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43617 } }
I20260811 22:24:55.121299 3211 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:55.121357 3211 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8e66805e45c4514981560743a8d458f, State: Initialized, Role: FOLLOWER
I20260811 22:24:55.121795 3214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:55.120321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.56:56441 (local address 127.0.0.1:43617)
0811 22:24:55.120438 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:55.120443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:55.120617 (+ 174us) server_negotiation.cc:407] Connection header received
0811 22:24:55.120716 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:55.120719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:55.120756 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:55.120835 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:55.121437 (+ 602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:55.121440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:55.121443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:55.121455 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:55.121485 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:55.121498 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:55.121585 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:55.121639 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 22:24:55.121679 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:24:55.121886 3211 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e66805e45c4514981560743a8d458f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43617 } }
I20260811 22:24:55.122035 3211 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:55.122102 3211 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:55.122206 3211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:55.123334 3211 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e66805e45c4514981560743a8d458f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43617 } }
I20260811 22:24:55.123658 3211 leader_election.cc:302] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8e66805e45c4514981560743a8d458f; no voters:
I20260811 22:24:55.123921 3211 leader_election.cc:288] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:55.124049 3225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:55.124281 3225 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [term 1 LEADER]: Becoming Leader. State: Replica: b8e66805e45c4514981560743a8d458f, State: Running, Role: LEADER
I20260811 22:24:55.124626 3225 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e66805e45c4514981560743a8d458f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43617 } }
I20260811 22:24:55.125320 3211 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:55.125982 3225 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8e66805e45c4514981560743a8d458f. Latest consensus state: current_term: 1 leader_uuid: "b8e66805e45c4514981560743a8d458f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e66805e45c4514981560743a8d458f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43617 } } }
I20260811 22:24:55.126107 3225 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:55.126441 3226 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8e66805e45c4514981560743a8d458f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8e66805e45c4514981560743a8d458f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43617 } } }
I20260811 22:24:55.126591 3226 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:55.126681 3237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:55.129981 3237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:55.136623 3237 catalog_manager.cc:1347] Generated new cluster ID: afdaa927b48641b0ba6ab4718b88de3d
I20260811 22:24:55.136726 3237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:55.150812 3237 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:55.151204 3237 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:55.157456 3237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8e66805e45c4514981560743a8d458f: Generated new TSK 0
I20260811 22:24:55.158244 3237 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:55.163457 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3075
2026-08-11T22:24:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1667 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:55 dist-test-slave-4tdj krb5kdc[3264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:55 dist-test-slave-4tdj krb5kdc[3264](info): set up 2 sockets
Aug 11 22:24:55 dist-test-slave-4tdj krb5kdc[3264](info): commencing operation
krb5kdc: starting...
W20260811 22:24:56.389420 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:56 dist-test-slave-4tdj krb5kdc[3264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:24:56.522157 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35701
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:35701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:56.639067 3292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:56.639366 3292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:56.639429 3292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:56.643532 3292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:56.643618 3292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:56.643646 3292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:56.643667 3292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:56.643688 3292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:56.649403 3292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:35701
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:35701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:56.650714 3292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:56.651892 3292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:56.660799 3301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:56.660791 3302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:56.660805 3309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:56.661158 3292 server_base.cc:1034] running on GCE node
I20260811 22:24:56.661782 3292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:56.663074 3292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:56.664258 3292 hybrid_clock.cc:648] HybridClock initialized: now 1786487096664231 us; error 42 us; skew 500 ppm
Aug 11 22:24:56 dist-test-slave-4tdj krb5kdc[3264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487096, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:56.667219 3292 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:56.668274 3292 webserver.cc:460] Webserver started at http://127.29.3.126:34331/ using document root <none> and password file <none>
I20260811 22:24:56.668883 3292 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:56.668954 3292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:56.669165 3292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:56.671409 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "3922951873f6459daa8178dc5b1810c9"
format_stamp: "Formatted at 2026-08-11 22:24:56 on dist-test-slave-4tdj"
server_key: "4806c15396731eceddd3473e55d90928"
server_key_iv: "0fc1e52303df96257b10fe9a2cee774f"
server_key_version: "encryptionkey@0"
I20260811 22:24:56.671998 3292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "3922951873f6459daa8178dc5b1810c9"
format_stamp: "Formatted at 2026-08-11 22:24:56 on dist-test-slave-4tdj"
server_key: "4806c15396731eceddd3473e55d90928"
server_key_iv: "0fc1e52303df96257b10fe9a2cee774f"
server_key_version: "encryptionkey@0"
I20260811 22:24:56.676020 3292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:24:56.678725 3325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:56.680121 3292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:56.680266 3292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "3922951873f6459daa8178dc5b1810c9"
format_stamp: "Formatted at 2026-08-11 22:24:56 on dist-test-slave-4tdj"
server_key: "4806c15396731eceddd3473e55d90928"
server_key_iv: "0fc1e52303df96257b10fe9a2cee774f"
server_key_version: "encryptionkey@0"
I20260811 22:24:56.680387 3292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:56.703945 3292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:56.707628 3292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:56.707842 3292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:56.715276 3292 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:35701
I20260811 22:24:56.715353 3430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:35701 every 8 connection(s)
I20260811 22:24:56.716336 3292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:56.719388 3434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:56.720108 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3292
I20260811 22:24:56.720263 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:56.720597 29709 external_mini_cluster.cc:1428] Setting key 622ceb79bc5934e4f7f96d147ff32302
I20260811 22:24:56.726351 3434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9: Bootstrap starting.
I20260811 22:24:56.728917 3434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:56.729766 3434 log.cc:824] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9: Log is configured to *not* fsync() on all Append() calls
Aug 11 22:24:56 dist-test-slave-4tdj krb5kdc[3264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:24:56.732710 3434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9: No bootstrap required, opened a new log
I20260811 22:24:56.736151 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:56.722158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56465 (local address 127.29.3.126:35701)
0811 22:24:56.722779 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:56.722796 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:56.722851 (+ 55us) server_negotiation.cc:407] Connection header received
0811 22:24:56.723564 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:56.723584 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:56.723967 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:56.724330 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:56.725211 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:56.726247 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:56.727216 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:56.727728 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:56.727892 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:56.731094 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:56.731101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:56.731118 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:56.731151 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:56.732962 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:56.733540 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:56.733546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:56.733551 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:56.733687 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:56.734061 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:56.734065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:56.734067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:56.734572 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:56.734781 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:56.735118 (+ 337us) server_negotiation.cc:299] Negotiation successful
0811 22:24:56.735354 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":175,"threads_started":1}
I20260811 22:24:56.736738 3434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3922951873f6459daa8178dc5b1810c9" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35701 } }
I20260811 22:24:56.737043 3434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:56.737113 3434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3922951873f6459daa8178dc5b1810c9, State: Initialized, Role: FOLLOWER
I20260811 22:24:56.737596 3434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3922951873f6459daa8178dc5b1810c9" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35701 } }
I20260811 22:24:56.737736 3434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:56.737798 3434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:56.737895 3434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:56.739094 3434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3922951873f6459daa8178dc5b1810c9" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35701 } }
I20260811 22:24:56.739487 3434 leader_election.cc:302] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3922951873f6459daa8178dc5b1810c9; no voters:
I20260811 22:24:56.739578 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:56.739835 3434 leader_election.cc:288] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:56.739933 3440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:56.740142 3440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [term 1 LEADER]: Becoming Leader. State: Replica: 3922951873f6459daa8178dc5b1810c9, State: Running, Role: LEADER
I20260811 22:24:56.740481 3440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3922951873f6459daa8178dc5b1810c9" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35701 } }
I20260811 22:24:56.741163 3434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:56.742036 3445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3922951873f6459daa8178dc5b1810c9. Latest consensus state: current_term: 1 leader_uuid: "3922951873f6459daa8178dc5b1810c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3922951873f6459daa8178dc5b1810c9" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35701 } } }
I20260811 22:24:56.742219 3445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:56.742666 3460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:56.742784 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3922951873f6459daa8178dc5b1810c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3922951873f6459daa8178dc5b1810c9" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 35701 } } }
I20260811 22:24:56.742895 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:56.746519 3460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:56.752908 3437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:56.742109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37533 (local address 127.29.3.126:35701)
0811 22:24:56.742252 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:56.742257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:56.743906 (+ 1649us) server_negotiation.cc:407] Connection header received
0811 22:24:56.744034 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:56.744037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:56.744090 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:56.744157 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:56.744993 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:56.745763 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:56.747111 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:56.747458 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:56.747570 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:56.749325 (+ 1755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:56.749329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:56.749332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:24:56.749360 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:56.751009 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:56.751518 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:56.751522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:56.751523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:56.751620 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:24:56.752020 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:24:56.752024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:24:56.752026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:24:56.752229 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:56.752367 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:56.752653 (+ 286us) server_negotiation.cc:299] Negotiation successful
0811 22:24:56.752751 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 22:24:56.753638 3460 catalog_manager.cc:1347] Generated new cluster ID: ccba675333174de39b73cecd9342e2b1
I20260811 22:24:56.753705 3460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:56.769416 3460 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:56.770869 3460 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:56.777683 3460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3922951873f6459daa8178dc5b1810c9: Generated new TSK 0
I20260811 22:24:56.778663 3460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:56.791052 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3292
W20260811 22:24:56.810370 3262 connection.cc:535] client connection to 127.29.3.126:35701 recv error: Network error: failed to read from TLS socket (remote: 127.29.3.126:35701): Connection reset by peer (error 104)
2026-08-11T22:24:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1623 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:56 dist-test-slave-4tdj krb5kdc[3480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:56 dist-test-slave-4tdj krb5kdc[3480](info): set up 2 sockets
Aug 11 22:24:56 dist-test-slave-4tdj krb5kdc[3480](info): commencing operation
krb5kdc: starting...
W20260811 22:24:58.008824 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:58 dist-test-slave-4tdj krb5kdc[3480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:24:58.139460 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40137
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:40137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:58.260363 3498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:58.260692 3498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:58.260739 3498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:58.264832 3498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:58.264920 3498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:58.264941 3498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:58.264961 3498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:58.264978 3498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:58.270752 3498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:40137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:58.271901 3498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:58.272967 3498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:58.281975 3510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:58.281989 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:58.282030 3498 server_base.cc:1034] running on GCE node
W20260811 22:24:58.281999 3506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:58.282819 3498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:58.284026 3498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:58.285188 3498 hybrid_clock.cc:648] HybridClock initialized: now 1786487098285164 us; error 47 us; skew 500 ppm
Aug 11 22:24:58 dist-test-slave-4tdj krb5kdc[3480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487098, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:58.288199 3498 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:58.289149 3498 webserver.cc:460] Webserver started at http://127.29.3.126:42607/ using document root <none> and password file <none>
I20260811 22:24:58.289722 3498 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:58.289788 3498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:58.289968 3498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:58.292198 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "78b2f84e03cb4013a50bdb0947a83040"
format_stamp: "Formatted at 2026-08-11 22:24:58 on dist-test-slave-4tdj"
server_key: "5e3dad7b404f094b4a0ae37df8f5a579"
server_key_iv: "4ce497b8a037f205607f72eaeeaa60f9"
server_key_version: "encryptionkey@0"
I20260811 22:24:58.292773 3498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "78b2f84e03cb4013a50bdb0947a83040"
format_stamp: "Formatted at 2026-08-11 22:24:58 on dist-test-slave-4tdj"
server_key: "5e3dad7b404f094b4a0ae37df8f5a579"
server_key_iv: "4ce497b8a037f205607f72eaeeaa60f9"
server_key_version: "encryptionkey@0"
I20260811 22:24:58.296819 3498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:24:58.299436 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:58.300781 3498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:24:58.300915 3498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "78b2f84e03cb4013a50bdb0947a83040"
format_stamp: "Formatted at 2026-08-11 22:24:58 on dist-test-slave-4tdj"
server_key: "5e3dad7b404f094b4a0ae37df8f5a579"
server_key_iv: "4ce497b8a037f205607f72eaeeaa60f9"
server_key_version: "encryptionkey@0"
I20260811 22:24:58.301025 3498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:58.321074 3498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:58.324568 3498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:58.324774 3498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:58.332130 3498 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:40137
I20260811 22:24:58.332136 3636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:40137 every 8 connection(s)
I20260811 22:24:58.333181 3498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:58.336257 3637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:58.337695 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3498
I20260811 22:24:58.337852 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:58.338160 29709 external_mini_cluster.cc:1428] Setting key 741787516a6523616020c957d2df8f53
I20260811 22:24:58.342897 3637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040: Bootstrap starting.
I20260811 22:24:58.345750 3637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:58.346642 3637 log.cc:824] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:58.347661 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:58.339619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46627 (local address 127.29.3.126:40137)
0811 22:24:58.340054 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:58.340071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:58.340109 (+ 38us) server_negotiation.cc:407] Connection header received
0811 22:24:58.340777 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:58.340795 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:58.341167 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:58.341468 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:58.341867 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:58.342924 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:58.344122 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:58.344622 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:58.344752 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:58.345734 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:58.345740 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:58.345755 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:58.346228 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:58.346346 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:58.346382 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:58.346502 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:58.346687 (+ 185us) server_negotiation.cc:299] Negotiation successful
0811 22:24:58.346837 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260811 22:24:58.348964 3637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040: No bootstrap required, opened a new log
I20260811 22:24:58.350838 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:58.353056 3637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b2f84e03cb4013a50bdb0947a83040" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40137 } }
I20260811 22:24:58.353331 3637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:58.353404 3637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b2f84e03cb4013a50bdb0947a83040, State: Initialized, Role: FOLLOWER
I20260811 22:24:58.353850 3637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b2f84e03cb4013a50bdb0947a83040" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40137 } }
I20260811 22:24:58.353988 3637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:58.354060 3637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:58.354164 3637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:58.355165 3637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b2f84e03cb4013a50bdb0947a83040" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40137 } }
I20260811 22:24:58.355520 3637 leader_election.cc:302] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78b2f84e03cb4013a50bdb0947a83040; no voters:
I20260811 22:24:58.355875 3637 leader_election.cc:288] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:58.355949 3657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:58.356161 3657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [term 1 LEADER]: Becoming Leader. State: Replica: 78b2f84e03cb4013a50bdb0947a83040, State: Running, Role: LEADER
I20260811 22:24:58.356480 3657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b2f84e03cb4013a50bdb0947a83040" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40137 } }
I20260811 22:24:58.357303 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:58.352830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59589 (local address 127.29.3.126:40137)
0811 22:24:58.352949 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:58.352954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:58.353126 (+ 172us) server_negotiation.cc:407] Connection header received
0811 22:24:58.353281 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:58.353284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:58.353327 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:58.353388 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:58.353769 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:58.354444 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:58.355240 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:24:58.355554 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:24:58.355646 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:24:58.356859 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:58.356862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:58.356866 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:58.356879 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:58.356905 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:58.356916 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:58.356982 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:58.357133 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 22:24:58.357169 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:24:58.357324 3637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:58.358513 3658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "78b2f84e03cb4013a50bdb0947a83040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b2f84e03cb4013a50bdb0947a83040" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40137 } } }
I20260811 22:24:58.358654 3658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:58.359002 3659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 78b2f84e03cb4013a50bdb0947a83040. Latest consensus state: current_term: 1 leader_uuid: "78b2f84e03cb4013a50bdb0947a83040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78b2f84e03cb4013a50bdb0947a83040" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 40137 } } }
I20260811 22:24:58.359068 3676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:58.359102 3659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:58.362215 3676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:58.367552 3676 catalog_manager.cc:1347] Generated new cluster ID: b669e1649b074e0a9d8a647fd0c73b7d
I20260811 22:24:58.367637 3676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:58.403415 3676 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:58.404822 3676 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:58.414490 3676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 78b2f84e03cb4013a50bdb0947a83040: Generated new TSK 0
I20260811 22:24:58.415422 3676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:58.430217 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3498
W20260811 22:24:58.441625 3476 connection.cc:535] client connection to 127.29.3.126:40137 recv error: Network error: failed to read from TLS socket (remote: 127.29.3.126:40137): Connection reset by peer (error 104)
2026-08-11T22:24:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1631 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:24:58 dist-test-slave-4tdj krb5kdc[3692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:24:58 dist-test-slave-4tdj krb5kdc[3692](info): set up 2 sockets
Aug 11 22:24:58 dist-test-slave-4tdj krb5kdc[3692](info): commencing operation
krb5kdc: starting...
W20260811 22:24:59.624760 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:24:59 dist-test-slave-4tdj krb5kdc[3692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:24:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:24:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:24:59.759387 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:46835
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:46835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:24:59.877180 3717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:24:59.877489 3717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:24:59.877533 3717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:24:59.881697 3717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:24:59.881788 3717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:24:59.881807 3717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:24:59.881822 3717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:24:59.881836 3717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:24:59.887506 3717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:46835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:46835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:24:59.888657 3717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:24:59.889679 3717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:24:59.898476 3737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:59.898566 3734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:24:59.898496 3735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:24:59.898864 3717 server_base.cc:1034] running on GCE node
I20260811 22:24:59.899495 3717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:24:59.900660 3717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:24:59.901829 3717 hybrid_clock.cc:648] HybridClock initialized: now 1786487099901815 us; error 34 us; skew 500 ppm
Aug 11 22:24:59 dist-test-slave-4tdj krb5kdc[3692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487099, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:24:59.904659 3717 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:24:59.905582 3717 webserver.cc:460] Webserver started at http://127.29.3.126:36417/ using document root <none> and password file <none>
I20260811 22:24:59.906133 3717 fs_manager.cc:359] Metadata directory not provided
I20260811 22:24:59.906193 3717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:24:59.906425 3717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:24:59.908586 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "3b40eaf2ed48429d8192acc6291a0a91"
format_stamp: "Formatted at 2026-08-11 22:24:59 on dist-test-slave-4tdj"
server_key: "161ed2964c54d0a47244db0cf726d5c9"
server_key_iv: "a5c65f8e0501196f094f7a9921e19d38"
server_key_version: "encryptionkey@0"
I20260811 22:24:59.909133 3717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "3b40eaf2ed48429d8192acc6291a0a91"
format_stamp: "Formatted at 2026-08-11 22:24:59 on dist-test-slave-4tdj"
server_key: "161ed2964c54d0a47244db0cf726d5c9"
server_key_iv: "a5c65f8e0501196f094f7a9921e19d38"
server_key_version: "encryptionkey@0"
I20260811 22:24:59.913008 3717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 22:24:59.915635 3746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:24:59.916972 3717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:24:59.917088 3717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "3b40eaf2ed48429d8192acc6291a0a91"
format_stamp: "Formatted at 2026-08-11 22:24:59 on dist-test-slave-4tdj"
server_key: "161ed2964c54d0a47244db0cf726d5c9"
server_key_iv: "a5c65f8e0501196f094f7a9921e19d38"
server_key_version: "encryptionkey@0"
I20260811 22:24:59.917207 3717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:24:59.921890 3717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:24:59.925024 3717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:24:59.925184 3717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:24:59.932557 3717 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:46835
I20260811 22:24:59.932598 3830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:46835 every 8 connection(s)
I20260811 22:24:59.933624 3717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:24:59.936867 3832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:24:59.937196 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3717
I20260811 22:24:59.937340 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:24:59.937669 29709 external_mini_cluster.cc:1428] Setting key 3c34f8bc667efa8e586ef126dd0cffe3
I20260811 22:24:59.943353 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:59.939107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46997 (local address 127.29.3.126:46835)
0811 22:24:59.939588 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:59.939604 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:59.939643 (+ 39us) server_negotiation.cc:407] Connection header received
0811 22:24:59.940457 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:59.940478 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:59.940811 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:59.941115 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:59.942021 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:59.942026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:59.942040 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:59.942051 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:59.942110 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:59.942130 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:59.942248 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:59.942406 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 22:24:59.942579 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":139,"threads_started":1}
I20260811 22:24:59.944365 3832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91: Bootstrap starting.
I20260811 22:24:59.946225 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:24:59.947009 3832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91: Neither blocks nor log segments found. Creating new log.
I20260811 22:24:59.948040 3832 log.cc:824] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91: Log is configured to *not* fsync() on all Append() calls
I20260811 22:24:59.950452 3832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91: No bootstrap required, opened a new log
I20260811 22:24:59.950667 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:24:59.948843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37175 (local address 127.29.3.126:46835)
0811 22:24:59.948982 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:24:59.948987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:24:59.949265 (+ 278us) server_negotiation.cc:407] Connection header received
0811 22:24:59.949419 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:24:59.949422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:24:59.949457 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:24:59.949524 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:24:59.950246 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:24:59.950250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:24:59.950253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:24:59.950265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:24:59.950320 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:24:59.950335 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:24:59.950417 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 22:24:59.950486 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 22:24:59.950531 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 22:24:59.954452 3832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40eaf2ed48429d8192acc6291a0a91" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 46835 } }
I20260811 22:24:59.954730 3832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:24:59.954793 3832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b40eaf2ed48429d8192acc6291a0a91, State: Initialized, Role: FOLLOWER
I20260811 22:24:59.955256 3832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40eaf2ed48429d8192acc6291a0a91" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 46835 } }
I20260811 22:24:59.955399 3832 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:24:59.955468 3832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:24:59.955574 3832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:24:59.956725 3832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40eaf2ed48429d8192acc6291a0a91" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 46835 } }
I20260811 22:24:59.957091 3832 leader_election.cc:302] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b40eaf2ed48429d8192acc6291a0a91; no voters:
I20260811 22:24:59.957464 3832 leader_election.cc:288] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:24:59.957614 3851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:24:59.957870 3851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [term 1 LEADER]: Becoming Leader. State: Replica: 3b40eaf2ed48429d8192acc6291a0a91, State: Running, Role: LEADER
I20260811 22:24:59.958196 3851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40eaf2ed48429d8192acc6291a0a91" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 46835 } }
I20260811 22:24:59.958778 3832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:24:59.959897 3856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b40eaf2ed48429d8192acc6291a0a91. Latest consensus state: current_term: 1 leader_uuid: "3b40eaf2ed48429d8192acc6291a0a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40eaf2ed48429d8192acc6291a0a91" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 46835 } } }
I20260811 22:24:59.960037 3856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:59.960227 3854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b40eaf2ed48429d8192acc6291a0a91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b40eaf2ed48429d8192acc6291a0a91" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 46835 } } }
I20260811 22:24:59.960381 3854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91 [sys.catalog]: This master's current role is: LEADER
I20260811 22:24:59.960467 3872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:24:59.963516 3872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:24:59.969666 3872 catalog_manager.cc:1347] Generated new cluster ID: f1ba58e5b7f44ba2b8d8a363bca2ebf6
I20260811 22:24:59.969775 3872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:24:59.978755 3872 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:24:59.979172 3872 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:24:59.988008 3872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b40eaf2ed48429d8192acc6291a0a91: Generated new TSK 0
I20260811 22:24:59.988963 3872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:24:59.997220 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3717
2026-08-11T22:25:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:00 dist-test-slave-4tdj krb5kdc[3892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:00 dist-test-slave-4tdj krb5kdc[3892](info): set up 2 sockets
Aug 11 22:25:00 dist-test-slave-4tdj krb5kdc[3892](info): commencing operation
krb5kdc: starting...
W20260811 22:25:01.218518 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:25:01 dist-test-slave-4tdj krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:25:01.355531 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:32987
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:40245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:32987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:01.471911 3916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:01.472236 3916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:01.472301 3916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:01.476444 3916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:01.476531 3916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:01.476567 3916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:01.476588 3916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:01.476610 3916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:01.482164 3916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:40245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:32987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:32987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:01.483352 3916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:01.484453 3916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:01.493428 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:01.493428 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:01.493433 3926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:01.493726 3916 server_base.cc:1034] running on GCE node
I20260811 22:25:01.494524 3916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:01.495728 3916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:01.496908 3916 hybrid_clock.cc:648] HybridClock initialized: now 1786487101496890 us; error 30 us; skew 500 ppm
Aug 11 22:25:01 dist-test-slave-4tdj krb5kdc[3892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487101, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:01.499809 3916 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:25:01.500795 3916 webserver.cc:460] Webserver started at http://127.29.3.126:40009/ using document root <none> and password file <none>
I20260811 22:25:01.501360 3916 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:01.501430 3916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:01.501662 3916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:01.503854 3916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "4677c00e52ba4a99999b18d5473e38c6"
format_stamp: "Formatted at 2026-08-11 22:25:01 on dist-test-slave-4tdj"
server_key: "e960d20db5944a637573e427ad2af465"
server_key_iv: "3a0db1dc90a14acbb16cac6865979246"
server_key_version: "encryptionkey@0"
I20260811 22:25:01.504433 3916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "4677c00e52ba4a99999b18d5473e38c6"
format_stamp: "Formatted at 2026-08-11 22:25:01 on dist-test-slave-4tdj"
server_key: "e960d20db5944a637573e427ad2af465"
server_key_iv: "3a0db1dc90a14acbb16cac6865979246"
server_key_version: "encryptionkey@0"
I20260811 22:25:01.508411 3916 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 22:25:01.511053 3951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:01.512351 3916 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 22:25:01.512487 3916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "4677c00e52ba4a99999b18d5473e38c6"
format_stamp: "Formatted at 2026-08-11 22:25:01 on dist-test-slave-4tdj"
server_key: "e960d20db5944a637573e427ad2af465"
server_key_iv: "3a0db1dc90a14acbb16cac6865979246"
server_key_version: "encryptionkey@0"
I20260811 22:25:01.512629 3916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:01.525193 3916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:01.528092 3916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:01.528299 3916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:01.535862 3916 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:32987
I20260811 22:25:01.535893 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:32987 every 8 connection(s)
I20260811 22:25:01.536895 3916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:01.540174 4043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:01.543301 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 3916
I20260811 22:25:01.543426 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:25:01.543701 29709 external_mini_cluster.cc:1428] Setting key c34af8279fbe60495f59ce0d8700de4f
I20260811 22:25:01.546765 4043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6: Bootstrap starting.
I20260811 22:25:01.549059 4043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:01.549856 4043 log.cc:824] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:01.551944 4043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6: No bootstrap required, opened a new log
Aug 11 22:25:01 dist-test-slave-4tdj krb5kdc[3892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:01.556125 4043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4677c00e52ba4a99999b18d5473e38c6" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32987 } }
I20260811 22:25:01.556396 4043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:01.556437 4043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4677c00e52ba4a99999b18d5473e38c6, State: Initialized, Role: FOLLOWER
I20260811 22:25:01.556982 4043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4677c00e52ba4a99999b18d5473e38c6" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32987 } }
I20260811 22:25:01.557189 4043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:01.557264 4043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:01.557363 4043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:01.558626 4043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4677c00e52ba4a99999b18d5473e38c6" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32987 } }
I20260811 22:25:01.559008 4043 leader_election.cc:302] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4677c00e52ba4a99999b18d5473e38c6; no voters:
I20260811 22:25:01.559373 4043 leader_election.cc:288] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:01.559491 4049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:01.559500 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:01.545193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38345 (local address 127.29.3.126:32987)
0811 22:25:01.545734 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:01.545756 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:01.545796 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:25:01.546492 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:01.546511 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:01.546873 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:01.547232 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:01.548039 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:01.549057 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:01.549915 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:01.550469 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:01.550620 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:01.553362 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:01.553369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:01.553388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:01.553424 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:01.556031 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:01.556495 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:01.556502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:01.556509 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:01.556671 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:01.556976 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:01.556979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:01.556980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:01.557589 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:01.557790 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:01.558075 (+ 285us) server_negotiation.cc:299] Negotiation successful
0811 22:25:01.558439 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":175,"threads_started":1}
I20260811 22:25:01.559726 4049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [term 1 LEADER]: Becoming Leader. State: Replica: 4677c00e52ba4a99999b18d5473e38c6, State: Running, Role: LEADER
I20260811 22:25:01.560101 4049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4677c00e52ba4a99999b18d5473e38c6" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32987 } }
I20260811 22:25:01.560738 4043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:01.561357 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4677c00e52ba4a99999b18d5473e38c6. Latest consensus state: current_term: 1 leader_uuid: "4677c00e52ba4a99999b18d5473e38c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4677c00e52ba4a99999b18d5473e38c6" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32987 } } }
I20260811 22:25:01.561486 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:01.561781 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4677c00e52ba4a99999b18d5473e38c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4677c00e52ba4a99999b18d5473e38c6" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32987 } } }
I20260811 22:25:01.561946 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:01.562466 4058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:01.562985 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:01.565968 4058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:01.571805 4058 catalog_manager.cc:1347] Generated new cluster ID: 5c5f2582863e4637bfd0e9cb71c9b24b
I20260811 22:25:01.571878 4058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:01.575311 4046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:01.565242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46205 (local address 127.29.3.126:32987)
0811 22:25:01.565367 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:01.565372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:01.565797 (+ 425us) server_negotiation.cc:407] Connection header received
0811 22:25:01.566396 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:01.566400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:01.566452 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:01.566506 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:01.567619 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:01.568473 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:01.569672 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:01.569924 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:01.570034 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:01.571844 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:01.571848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:01.571851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:01.571880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:01.573547 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:01.574025 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:01.574028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:01.574030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:01.574132 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:01.574402 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:01.574405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:01.574407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:01.574639 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:01.574766 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:01.574998 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 22:25:01.575142 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260811 22:25:01.582379 4058 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:01.583374 4058 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:01.589603 4058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4677c00e52ba4a99999b18d5473e38c6: Generated new TSK 0
I20260811 22:25:01.590420 4058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:01.594283 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 3916
2026-08-11T22:25:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:01 dist-test-slave-4tdj krb5kdc[4100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:01 dist-test-slave-4tdj krb5kdc[4100](info): set up 2 sockets
Aug 11 22:25:01 dist-test-slave-4tdj krb5kdc[4100](info): commencing operation
krb5kdc: starting...
W20260811 22:25:02.803731 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:25:02 dist-test-slave-4tdj krb5kdc[4100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:25:02.941462 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:32915
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:32915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:03.061900 4120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:03.062286 4120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:03.062398 4120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:03.066599 4120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:03.066679 4120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:03.066697 4120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:03.066713 4120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:03.066727 4120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:03.072444 4120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:32915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:32915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:03.073691 4120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:03.074815 4120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:03.083765 4133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:03.083835 4128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:03.083974 4120 server_base.cc:1034] running on GCE node
W20260811 22:25:03.083900 4127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:03.084609 4120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:03.085810 4120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:03.086977 4120 hybrid_clock.cc:648] HybridClock initialized: now 1786487103086961 us; error 28 us; skew 500 ppm
Aug 11 22:25:03 dist-test-slave-4tdj krb5kdc[4100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487103, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:03.089778 4120 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:25:03.090734 4120 webserver.cc:460] Webserver started at http://127.29.3.126:35035/ using document root <none> and password file <none>
I20260811 22:25:03.091280 4120 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:03.091341 4120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:03.091513 4120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:03.093675 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "3e28f562767d485e8bf4229109494297"
format_stamp: "Formatted at 2026-08-11 22:25:03 on dist-test-slave-4tdj"
server_key: "eec32ee1515e4d061de5c5d47e098b29"
server_key_iv: "2157c4c2842c518d83cf0d980355238d"
server_key_version: "encryptionkey@0"
I20260811 22:25:03.094234 4120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "3e28f562767d485e8bf4229109494297"
format_stamp: "Formatted at 2026-08-11 22:25:03 on dist-test-slave-4tdj"
server_key: "eec32ee1515e4d061de5c5d47e098b29"
server_key_iv: "2157c4c2842c518d83cf0d980355238d"
server_key_version: "encryptionkey@0"
I20260811 22:25:03.098199 4120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:25:03.100858 4152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:03.102119 4120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 22:25:03.102237 4120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "3e28f562767d485e8bf4229109494297"
format_stamp: "Formatted at 2026-08-11 22:25:03 on dist-test-slave-4tdj"
server_key: "eec32ee1515e4d061de5c5d47e098b29"
server_key_iv: "2157c4c2842c518d83cf0d980355238d"
server_key_version: "encryptionkey@0"
I20260811 22:25:03.102416 4120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:03.128455 4120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:03.131989 4120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:03.132192 4120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:03.139475 4120 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:32915
I20260811 22:25:03.139492 4244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:32915 every 8 connection(s)
I20260811 22:25:03.140480 4120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:03.143484 4246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:03.149616 4246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297: Bootstrap starting.
I20260811 22:25:03.149866 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4120
I20260811 22:25:03.150022 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:25:03.150363 29709 external_mini_cluster.cc:1428] Setting key c4e904cb7b74672c37cfeffe5423a103
I20260811 22:25:03.152664 4246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:03.153475 4246 log.cc:824] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:03.155565 4246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297: No bootstrap required, opened a new log
I20260811 22:25:03.159457 4246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e28f562767d485e8bf4229109494297" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32915 } }
I20260811 22:25:03.159710 4246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:03.159770 4246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e28f562767d485e8bf4229109494297, State: Initialized, Role: FOLLOWER
I20260811 22:25:03.159920 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:03.151828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51621 (local address 127.29.3.126:32915)
0811 22:25:03.152332 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:03.152348 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:03.152389 (+ 41us) server_negotiation.cc:407] Connection header received
0811 22:25:03.153164 (+ 775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:03.153188 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:03.153580 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:03.153944 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:03.154437 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:03.155454 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:03.156398 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:03.156891 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:03.157044 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:03.157994 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:03.157999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:03.158014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:03.158504 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:03.158566 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:03.158592 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:03.158716 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:03.158859 (+ 143us) server_negotiation.cc:299] Negotiation successful
0811 22:25:03.159083 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":152,"threads_started":1}
I20260811 22:25:03.160265 4246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e28f562767d485e8bf4229109494297" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32915 } }
I20260811 22:25:03.160434 4246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:03.160497 4246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:03.160594 4246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:03.161638 4246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e28f562767d485e8bf4229109494297" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32915 } }
I20260811 22:25:03.162015 4246 leader_election.cc:302] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e28f562767d485e8bf4229109494297; no voters:
I20260811 22:25:03.162482 4246 leader_election.cc:288] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:03.162552 4254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:03.162796 4254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [term 1 LEADER]: Becoming Leader. State: Replica: 3e28f562767d485e8bf4229109494297, State: Running, Role: LEADER
I20260811 22:25:03.163029 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:03.163219 4254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e28f562767d485e8bf4229109494297" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32915 } }
I20260811 22:25:03.163885 4246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:03.164906 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e28f562767d485e8bf4229109494297. Latest consensus state: current_term: 1 leader_uuid: "3e28f562767d485e8bf4229109494297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e28f562767d485e8bf4229109494297" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32915 } } }
I20260811 22:25:03.165040 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:03.165307 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e28f562767d485e8bf4229109494297" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e28f562767d485e8bf4229109494297" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32915 } } }
I20260811 22:25:03.165386 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:03.165757 4283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:03.169349 4283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:03.170244 4250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:03.165580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37729 (local address 127.29.3.126:32915)
0811 22:25:03.165699 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:03.165704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:03.166855 (+ 1151us) server_negotiation.cc:407] Connection header received
0811 22:25:03.166962 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:03.166965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:03.167019 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:03.167104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:03.167351 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:03.167877 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:03.168692 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:03.168917 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:03.169005 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:03.169821 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:03.169824 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:03.169826 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:03.169839 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:03.169866 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:03.169877 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:03.169937 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:03.170041 (+ 104us) server_negotiation.cc:299] Negotiation successful
0811 22:25:03.170099 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 22:25:03.175395 4283 catalog_manager.cc:1347] Generated new cluster ID: 4b125b323be349edbdef87ade8eacf8a
I20260811 22:25:03.175485 4283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:03.196885 4283 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:03.198369 4283 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:03.205861 4283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e28f562767d485e8bf4229109494297: Generated new TSK 0
I20260811 22:25:03.206902 4283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:03.218761 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4120
2026-08-11T22:25:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1620 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:03 dist-test-slave-4tdj krb5kdc[4302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:03 dist-test-slave-4tdj krb5kdc[4302](info): set up 2 sockets
Aug 11 22:25:03 dist-test-slave-4tdj krb5kdc[4302](info): commencing operation
krb5kdc: starting...
W20260811 22:25:04.443604 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:25:04 dist-test-slave-4tdj krb5kdc[4302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:25:04.579025 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:32885
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:32885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:04.698715 4323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:04.699049 4323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:04.699115 4323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:04.703809 4323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:04.703902 4323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:04.703929 4323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:04.703953 4323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:04.703981 4323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:04.710220 4323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:32885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:32885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:04.711452 4323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:04.712540 4323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:04.721406 4332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:04.721418 4333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:04.721530 4337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:04.721835 4323 server_base.cc:1034] running on GCE node
I20260811 22:25:04.722589 4323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:04.723775 4323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:04.724946 4323 hybrid_clock.cc:648] HybridClock initialized: now 1786487104724930 us; error 36 us; skew 500 ppm
Aug 11 22:25:04 dist-test-slave-4tdj krb5kdc[4302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487104, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:04.732864 4323 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:25:04.733928 4323 webserver.cc:460] Webserver started at http://127.29.3.126:33227/ using document root <none> and password file <none>
I20260811 22:25:04.734763 4323 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:04.734859 4323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:04.735107 4323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:04.737375 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "8bc1bddc602f48df82a57e9235a4027e"
format_stamp: "Formatted at 2026-08-11 22:25:04 on dist-test-slave-4tdj"
server_key: "68c75d16fc8cea1ff32482aa967bdbfe"
server_key_iv: "510d4693d058fc2c3bcc2f730174a83d"
server_key_version: "encryptionkey@0"
I20260811 22:25:04.737983 4323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "8bc1bddc602f48df82a57e9235a4027e"
format_stamp: "Formatted at 2026-08-11 22:25:04 on dist-test-slave-4tdj"
server_key: "68c75d16fc8cea1ff32482aa967bdbfe"
server_key_iv: "510d4693d058fc2c3bcc2f730174a83d"
server_key_version: "encryptionkey@0"
I20260811 22:25:04.742132 4323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 22:25:04.744947 4347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:04.746420 4323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:04.746585 4323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "8bc1bddc602f48df82a57e9235a4027e"
format_stamp: "Formatted at 2026-08-11 22:25:04 on dist-test-slave-4tdj"
server_key: "68c75d16fc8cea1ff32482aa967bdbfe"
server_key_iv: "510d4693d058fc2c3bcc2f730174a83d"
server_key_version: "encryptionkey@0"
I20260811 22:25:04.746709 4323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:04.751587 4323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:04.754949 4323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:04.755141 4323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:04.762429 4323 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:32885
I20260811 22:25:04.762459 4440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:32885 every 8 connection(s)
I20260811 22:25:04.763504 4323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:04.766670 4441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:04.767246 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4323
I20260811 22:25:04.767398 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:25:04.767717 29709 external_mini_cluster.cc:1428] Setting key 42ed773cd6a6c035d90ea880bc51f1d4
I20260811 22:25:04.773556 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:04.769295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38627 (local address 127.29.3.126:32885)
0811 22:25:04.769899 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:04.769919 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:04.769962 (+ 43us) server_negotiation.cc:407] Connection header received
0811 22:25:04.770736 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:04.770759 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:04.771082 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:04.771424 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:04.772272 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:04.772278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:04.772291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:04.772302 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:04.772363 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:04.772384 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:04.772520 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:04.772605 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 22:25:04.772777 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":167,"threads_started":1}
I20260811 22:25:04.773706 4441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e: Bootstrap starting.
I20260811 22:25:04.776618 4441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:04.777036 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:04.777545 4441 log.cc:824] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:04.780010 4441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e: No bootstrap required, opened a new log
I20260811 22:25:04.780893 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:04.779242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45553 (local address 127.29.3.126:32885)
0811 22:25:04.779375 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:04.779380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:04.779599 (+ 219us) server_negotiation.cc:407] Connection header received
0811 22:25:04.779703 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:04.779706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:04.779749 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:04.779839 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:04.780530 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:04.780533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:04.780536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 22:25:04.780549 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:04.780581 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 22:25:04.780594 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:04.780687 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:04.780725 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 22:25:04.780765 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:25:04.784353 4441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc1bddc602f48df82a57e9235a4027e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32885 } }
I20260811 22:25:04.784621 4441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 22:25:04.784672 4441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bc1bddc602f48df82a57e9235a4027e, State: Initialized, Role: FOLLOWER
I20260811 22:25:04.785130 4441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc1bddc602f48df82a57e9235a4027e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32885 } }
I20260811 22:25:04.785295 4441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 22:25:04.785364 4441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 22:25:04.785465 4441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:04.786510 4441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc1bddc602f48df82a57e9235a4027e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32885 } }
I20260811 22:25:04.786866 4441 leader_election.cc:302] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bc1bddc602f48df82a57e9235a4027e; no voters:
I20260811 22:25:04.787230 4441 leader_election.cc:288] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 22:25:04.787361 4459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:04.787573 4459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [term 1 LEADER]: Becoming Leader. State: Replica: 8bc1bddc602f48df82a57e9235a4027e, State: Running, Role: LEADER
I20260811 22:25:04.787946 4459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc1bddc602f48df82a57e9235a4027e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32885 } }
I20260811 22:25:04.788496 4441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:04.790064 4460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bc1bddc602f48df82a57e9235a4027e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc1bddc602f48df82a57e9235a4027e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32885 } } }
I20260811 22:25:04.790248 4460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:04.790056 4461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bc1bddc602f48df82a57e9235a4027e. Latest consensus state: current_term: 1 leader_uuid: "8bc1bddc602f48df82a57e9235a4027e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bc1bddc602f48df82a57e9235a4027e" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 32885 } } }
I20260811 22:25:04.790518 4461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:04.790724 4474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:04.794114 4474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:04.800513 4474 catalog_manager.cc:1347] Generated new cluster ID: 4ca3536209e74c1f88535a9d9d4520a1
I20260811 22:25:04.800611 4474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:04.813015 4474 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:04.813380 4474 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:04.828282 4474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bc1bddc602f48df82a57e9235a4027e: Generated new TSK 0
I20260811 22:25:04.829100 4474 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 22:25:04.839409 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4323
2026-08-11T22:25:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1616 ms)
[----------] 18 tests from AuthTokenIssuingTest (28943 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:04 dist-test-slave-4tdj krb5kdc[4498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:04 dist-test-slave-4tdj krb5kdc[4498](info): set up 2 sockets
Aug 11 22:25:04 dist-test-slave-4tdj krb5kdc[4498](info): commencing operation
krb5kdc: starting...
W20260811 22:25:06.038503 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.167s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:25:06.173084 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:33197
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:33197,127.29.3.125:37645,127.29.3.124:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:06.292140 4526 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:25:06.292505 4526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:06.292544 4526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:06.292610 4526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:06.297019 4526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:06.297118 4526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:06.297147 4526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:06.297169 4526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:06.297190 4526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:06.302973 4526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:33197,127.29.3.125:37645,127.29.3.124:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:33197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:06.304205 4526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:06.305392 4526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:06.314261 4539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:06.314261 4536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:06.314261 4533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:06.314519 4526 server_base.cc:1034] running on GCE node
I20260811 22:25:06.315274 4526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:06.316496 4526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:06.317700 4526 hybrid_clock.cc:648] HybridClock initialized: now 1786487106317661 us; error 55 us; skew 500 ppm
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:06.320683 4526 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:25:06.321679 4526 webserver.cc:460] Webserver started at http://127.29.3.126:43673/ using document root <none> and password file <none>
I20260811 22:25:06.322238 4526 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:06.322337 4526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:06.322570 4526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:06.324735 4526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "512f3ce6f7594218b766c7514c1426bd"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "45079ba89562d1ae8974d68618ee9780"
server_key_iv: "d504848b306c895bfebcd88d6061bcc2"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.325321 4526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "512f3ce6f7594218b766c7514c1426bd"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "45079ba89562d1ae8974d68618ee9780"
server_key_iv: "d504848b306c895bfebcd88d6061bcc2"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.329299 4526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 22:25:06.332103 4554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:06.333423 4526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 22:25:06.333555 4526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "512f3ce6f7594218b766c7514c1426bd"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "45079ba89562d1ae8974d68618ee9780"
server_key_iv: "d504848b306c895bfebcd88d6061bcc2"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.333678 4526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:06.343689 4526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:06.347057 4526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:06.347258 4526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:06.354600 4526 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:33197
I20260811 22:25:06.354876 4636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:33197 every 8 connection(s)
I20260811 22:25:06.355652 4526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:06.358618 4639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:06.361083 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4526
I20260811 22:25:06.361243 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:25:06.361532 29709 external_mini_cluster.cc:1428] Setting key 6f2db182bf48fb84a35efcac32c4bdaa
I20260811 22:25:06.363185 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.3.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.125@KRBTEST.COM" created.
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:06.379684 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.365750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.29.3.126:33197)
0811 22:25:06.366335 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.366355 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.366618 (+ 263us) server_negotiation.cc:407] Connection header received
0811 22:25:06.367907 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.367912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.368107 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.368282 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.369413 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.370358 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.371700 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.372129 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.372270 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.374693 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.374701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.374719 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.374747 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.376406 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.376882 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.376887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.376889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.376997 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.377294 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.377297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.377299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.377710 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.377840 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.378229 (+ 389us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.378513 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":374,"thread_start_us":157,"threads_started":1}
I20260811 22:25:06.379684 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.365750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:33197 (local address 127.0.0.1:48096)
0811 22:25:06.366435 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.366488 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.367529 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.368345 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.368352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.368854 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.369234 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.369256 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.370500 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.370504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.371576 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.371587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.372277 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.372280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.372376 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.372906 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.372924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.374521 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.376516 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.376525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.376538 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.376774 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.377086 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.377089 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.377091 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.377206 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.377823 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.377830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.377978 (+ 148us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.378142 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.378581 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":100,"spinlock_wait_cycles":57600,"thread_start_us":158,"threads_started":1}
I20260811 22:25:06.382483 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.383323 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.383018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:34966)
0811 22:25:06.383169 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.383249 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 22:25:06.383518 4559 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
I20260811 22:25:06.384375 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.384148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:34972)
0811 22:25:06.384257 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.384307 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 22:25:06.384507 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:37645: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab.
Entry for principal kudu/127.29.3.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab.
I20260811 22:25:06.390782 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:37645
--webserver_interface=127.29.3.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab
--principal=kudu/127.29.3.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:33197,127.29.3.125:37645,127.29.3.124:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:25:06.433781 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } attempt: 1
I20260811 22:25:06.434824 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.434388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:34982)
0811 22:25:06.434577 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.434715 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 22:25:06.435667 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.435391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:34994)
0811 22:25:06.435513 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.435573 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 22:25:06.435827 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:37645: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
I20260811 22:25:06.504046 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } attempt: 2
I20260811 22:25:06.505142 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.504689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:35000)
0811 22:25:06.504966 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.505046 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 22:25:06.505942 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.505742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:35016)
0811 22:25:06.505834 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.505882 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 22:25:06.506072 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:37645: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37645: connect: Connection refused (error 111)
W20260811 22:25:06.510108 4650 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:25:06.510465 4650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:06.510511 4650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:06.510576 4650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:06.514729 4650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:06.514789 4650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:06.514807 4650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:06.514828 4650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:06.514855 4650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:06.520380 4650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:33197,127.29.3.125:37645,127.29.3.124:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab
--principal=kudu/127.29.3.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:37645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.3.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:06.521531 4650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:06.522652 4650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 22:25:06.531368 4650 server_base.cc:1034] running on GCE node
W20260811 22:25:06.531428 4665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:06.531410 4661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:06.531463 4663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:06.532263 4650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:06.533390 4650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:06.534554 4650 hybrid_clock.cc:648] HybridClock initialized: now 1786487106534537 us; error 33 us; skew 500 ppm
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:06.537508 4650 init.cc:377] Logged in from keytab as kudu/127.29.3.125@KRBTEST.COM (short username kudu)
I20260811 22:25:06.538602 4650 webserver.cc:460] Webserver started at http://127.29.3.125:43663/ using document root <none> and password file <none>
I20260811 22:25:06.539187 4650 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:06.539264 4650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:06.539469 4650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:06.541643 4650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data/instance:
uuid: "b8351e8b72634b609d2eaf63e8b542bd"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "5cfb1f5613892b0057e2c9302aab94fd"
server_key_iv: "5ce60ca72ce3bcbbc96b11a30c7dad2f"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.542244 4650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal/instance:
uuid: "b8351e8b72634b609d2eaf63e8b542bd"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "5cfb1f5613892b0057e2c9302aab94fd"
server_key_iv: "5ce60ca72ce3bcbbc96b11a30c7dad2f"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.546191 4650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 22:25:06.548767 4678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:06.550134 4650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 22:25:06.550272 4650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal
uuid: "b8351e8b72634b609d2eaf63e8b542bd"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "5cfb1f5613892b0057e2c9302aab94fd"
server_key_iv: "5ce60ca72ce3bcbbc96b11a30c7dad2f"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.550429 4650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:06.568121 4650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:06.571326 4650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:06.571512 4650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:06.578980 4650 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.125:37645
I20260811 22:25:06.579272 4780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.125:37645 every 8 connection(s)
I20260811 22:25:06.580106 4650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
I20260811 22:25:06.583226 4781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:06.587507 4781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.588753 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4650
I20260811 22:25:06.588862 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-1/wal/instance
I20260811 22:25:06.589149 29709 external_mini_cluster.cc:1428] Setting key 76d1357c39a3012a7dc8e31a0081bed7
I20260811 22:25:06.597237 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } attempt: 3
WARNING: no policy specified for kudu/127.29.3.124@KRBTEST.COM; defaulting to no policy
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.124@KRBTEST.COM" created.
I20260811 22:25:06.602751 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.589648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.29.3.126:33197)
0811 22:25:06.589782 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.589785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.591066 (+ 1281us) server_negotiation.cc:407] Connection header received
0811 22:25:06.592371 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.592374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.592434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.592512 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.593878 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.594466 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.595642 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.595895 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.595989 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.598718 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.598722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.598724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.598753 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.600229 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.600857 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.600860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.600862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.600943 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.601348 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.601351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.601353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.601581 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.601654 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.602446 (+ 792us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.602577 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:25:06.603833 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.590230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:33197 (local address 127.0.0.1:48100)
0811 22:25:06.590886 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.590935 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.592079 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.592745 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.592755 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.593174 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.593711 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.593726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.594608 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.594611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.595510 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.595520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.596054 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.596059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.596244 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.596821 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.596839 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.598536 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.600347 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.600354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.600372 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.600758 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.601029 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.601032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.601034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.601232 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.601753 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.601759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.602098 (+ 339us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.602347 (+ 249us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.602729 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":186,"threads_started":1}
I20260811 22:25:06.605933 4781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.606886 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.597726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:35030)
0811 22:25:06.597939 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.597952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.598035 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.598738 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.598741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.598946 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.599075 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.599081 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.600057 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.600061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.600591 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.600600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.601221 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.601226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.601335 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.601774 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.601788 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.603217 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.605284 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.605287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.605289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.605474 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.605753 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.605756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.605758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.605836 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.606362 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.606365 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.606472 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.606576 (+ 104us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.606723 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"spinlock_wait_cycles":1280}
I20260811 22:25:06.607613 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.597862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.29.3.125:37645)
0811 22:25:06.598286 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.598323 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.598347 (+ 24us) server_negotiation.cc:407] Connection header received
0811 22:25:06.598391 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.598397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.598611 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.598751 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.599197 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.599933 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.600704 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.601104 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.601280 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.603372 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.603381 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.603394 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.603420 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.605146 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.605583 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.605588 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.605589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.605658 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.605916 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.605919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.605921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.606240 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.606419 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.607256 (+ 837us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.607418 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":113,"threads_started":1}
I20260811 22:25:06.609498 4639 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.610349 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.610039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46312)
0811 22:25:06.610169 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.610236 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 22:25:06.611081 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.610845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46318)
0811 22:25:06.610983 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.611028 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
W20260811 22:25:06.611267 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:36633: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
I20260811 22:25:06.615311 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.606496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:35046)
0811 22:25:06.606607 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.606620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.606692 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.607168 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.607171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.607346 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.607452 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.607457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.608284 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.608287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.608989 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.608999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.609586 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.609590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.609687 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.610137 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.610150 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.611626 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.613878 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.613882 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.613884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.614123 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.614401 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.614403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.614404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.614483 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.614861 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.614863 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.614946 (+ 83us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.615027 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.615147 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260811 22:25:06.615383 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.606691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.29.3.125:37645)
0811 22:25:06.606968 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.606976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.606990 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:25:06.607025 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.607028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.607067 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.607526 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.607573 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.608179 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.609113 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.609470 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.609575 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.611789 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.611793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.611795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.611823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.613776 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.614231 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.614234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.614235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.614332 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.614547 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.614549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.614550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.614786 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.614866 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.615069 (+ 203us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.615208 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260811 22:25:06.616010 4781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.616734 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.616491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46326)
0811 22:25:06.616591 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.616672 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 22:25:06.617030 4679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
I20260811 22:25:06.617913 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.617707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46340)
0811 22:25:06.617813 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.617860 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:25:06.618039 4781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:36633: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab.
Entry for principal kudu/127.29.3.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab.
I20260811 22:25:06.621753 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:36633
--webserver_interface=127.29.3.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab
--principal=kudu/127.29.3.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:33197,127.29.3.125:37645,127.29.3.124:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:25:06.642417 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } attempt: 1
I20260811 22:25:06.643224 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.642926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46346)
0811 22:25:06.643078 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.643138 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260811 22:25:06.643900 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.643688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46358)
0811 22:25:06.643791 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.643837 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 22:25:06.644016 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:36633: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
I20260811 22:25:06.667311 4781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } attempt: 1
I20260811 22:25:06.668259 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.667940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46372)
0811 22:25:06.668100 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.668164 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260811 22:25:06.668952 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.668742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46376)
0811 22:25:06.668849 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.668894 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:25:06.669067 4781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:36633: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
I20260811 22:25:06.719170 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } attempt: 2
I20260811 22:25:06.720085 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.719781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46384)
0811 22:25:06.719937 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.720003 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 22:25:06.720816 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.720600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46396)
0811 22:25:06.720708 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.720755 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260811 22:25:06.720938 4639 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:36633: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
I20260811 22:25:06.737244 4781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } attempt: 2
I20260811 22:25:06.738188 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.737826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46400)
0811 22:25:06.738013 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.738088 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 22:25:06.738922 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.738705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46416)
0811 22:25:06.738808 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.738862 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 22:25:06.739058 4781 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:36633: Network error: Client connection negotiation failed: client connection to 127.29.3.124:36633: connect: Connection refused (error 111)
W20260811 22:25:06.742802 4794 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:25:06.743103 4794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:06.743126 4794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:06.743165 4794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:06.747303 4794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:06.747375 4794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:06.747393 4794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:06.747409 4794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:06.747424 4794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:06.753180 4794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:33197,127.29.3.125:37645,127.29.3.124:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab
--principal=kudu/127.29.3.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.3.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:06.754410 4794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:06.755486 4794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:06.764326 4804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:06.764312 4801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:06.764421 4794 server_base.cc:1034] running on GCE node
W20260811 22:25:06.764312 4802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:06.765285 4794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:06.766518 4794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:06.767697 4794 hybrid_clock.cc:648] HybridClock initialized: now 1786487106767678 us; error 32 us; skew 500 ppm
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:06.770551 4794 init.cc:377] Logged in from keytab as kudu/127.29.3.124@KRBTEST.COM (short username kudu)
I20260811 22:25:06.771539 4794 webserver.cc:460] Webserver started at http://127.29.3.124:39437/ using document root <none> and password file <none>
I20260811 22:25:06.772092 4794 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:06.772157 4794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:06.772342 4794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:06.774727 4794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data/instance:
uuid: "f9623c23e62f4b30a2aaa09c8b077f9c"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "02503aab5d77c37701afc067563ef6bd"
server_key_iv: "488ae00d4b9b4aa2a9fabd3083260599"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.775300 4794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal/instance:
uuid: "f9623c23e62f4b30a2aaa09c8b077f9c"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "02503aab5d77c37701afc067563ef6bd"
server_key_iv: "488ae00d4b9b4aa2a9fabd3083260599"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.779378 4794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 22:25:06.782164 4812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:06.783828 4794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 22:25:06.783977 4794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal
uuid: "f9623c23e62f4b30a2aaa09c8b077f9c"
format_stamp: "Formatted at 2026-08-11 22:25:06 on dist-test-slave-4tdj"
server_key: "02503aab5d77c37701afc067563ef6bd"
server_key_iv: "488ae00d4b9b4aa2a9fabd3083260599"
server_key_version: "encryptionkey@0"
I20260811 22:25:06.784111 4794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:06.800846 4794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:06.804167 4794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:06.804363 4794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:06.811744 4794 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.124:36633
I20260811 22:25:06.812039 4900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.124:36633 every 8 connection(s)
I20260811 22:25:06.812844 4794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
I20260811 22:25:06.815905 4901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:06.819414 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 4794
I20260811 22:25:06.819537 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786486810933177-29709-0/minicluster-data/master-2/wal/instance
I20260811 22:25:06.819798 29709 external_mini_cluster.cc:1428] Setting key 287a1081775de95d2b85ea4d7c14dc97
I20260811 22:25:06.820096 4639 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } attempt: 3
I20260811 22:25:06.820111 4901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:25:06.830219 4781 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } attempt: 3
I20260811 22:25:06.833981 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.820556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46432)
0811 22:25:06.820773 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.820790 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.820892 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.823152 (+ 2260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.823155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.823388 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.823496 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.823504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.825199 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.825202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.826095 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.826108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.826951 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.827036 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.827538 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.827555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.828838 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.831437 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.831440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.831442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.831918 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.832531 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.832534 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.832536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.832635 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.833446 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.833449 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.833550 (+ 101us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.833645 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.833804 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260811 22:25:06.834239 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.822247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.29.3.126:33197)
0811 22:25:06.822398 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.822402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.822674 (+ 272us) server_negotiation.cc:407] Connection header received
0811 22:25:06.823119 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.823122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.823172 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.823244 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.824305 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.825050 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826107 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.826432 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826547 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.830098 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.830101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.830103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.830127 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.831817 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.832402 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.832406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.832408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.832482 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.832858 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.832861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.832863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.833185 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.833265 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.833791 (+ 526us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.833916 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260811 22:25:06.835788 4907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.821247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46426 (local address 127.29.3.124:36633)
0811 22:25:06.821720 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.821734 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.821768 (+ 34us) server_negotiation.cc:407] Connection header received
0811 22:25:06.823162 (+ 1394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.823165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.823244 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.823325 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.824602 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.825122 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826273 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.826848 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826942 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.830441 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.830444 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.830447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.830472 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.833199 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.833707 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.833711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.833712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.833790 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.834088 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.834091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.834093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.834340 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.834503 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.835193 (+ 690us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.835302 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":326,"spinlock_wait_cycles":41728,"thread_start_us":150,"threads_started":1}
I20260811 22:25:06.835898 4908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.821698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.29.3.124:36633)
0811 22:25:06.821957 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.821966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.821981 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:25:06.822623 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.822650 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.822953 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.823171 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.823782 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.825104 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826216 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.826840 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826932 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.828991 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.828999 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.829014 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.829038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.831319 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.832048 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.832055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.832057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.832208 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.832749 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.832752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.832754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.833269 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.833385 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.834259 (+ 874us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.834658 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":191,"spinlock_wait_cycles":21504,"thread_start_us":74,"threads_started":1}
I20260811 22:25:06.835791 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.822163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:33197 (local address 127.0.0.1:48110)
0811 22:25:06.822537 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.822587 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.822932 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.823412 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.823417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.823925 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.824144 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.824159 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.825185 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.825188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.825981 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.825999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.826593 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.826597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.826828 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.827445 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.827463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.829791 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.831948 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.831958 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.831970 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.832306 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.832577 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.832580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.832582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.832739 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.833346 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.833353 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.833522 (+ 169us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.834435 (+ 913us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.834739 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":114,"thread_start_us":90,"threads_started":1}
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:25:06.838553 4901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.840902 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.830820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.29.3.124:36633)
0811 22:25:06.831840 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.831849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.831863 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:25:06.831898 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.831901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.831943 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.832530 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.832638 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.833365 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.834873 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.835187 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.835649 (+ 462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.837428 (+ 1779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.837432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.837434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.837462 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.839297 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.839790 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.839793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.839794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.839868 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.840112 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.840114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.840115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.840291 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.840363 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.840667 (+ 304us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.840771 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":122,"threads_started":1}
I20260811 22:25:06.840919 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.830745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46446)
0811 22:25:06.831266 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.831284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.831371 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.832083 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.832086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.832291 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.832424 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.832431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.833481 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.833484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.834242 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.834253 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.835334 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.835338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.835424 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.835893 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.835908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.837304 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.839419 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.839423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.839425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.839684 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.839941 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.839944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.839946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.840039 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.840383 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.840386 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.840514 (+ 128us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.840623 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.840768 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 22:25:06.841576 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd: Bootstrap starting.
I20260811 22:25:06.844178 4781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: Bootstrap starting.
I20260811 22:25:06.844419 4639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:06.845242 4639 log.cc:824] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:06.846975 4781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: Neither blocks nor log segments found. Creating new log.
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:25:06.847934 4781 log.cc:824] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:06.848191 4639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd: No bootstrap required, opened a new log
I20260811 22:25:06.850029 4781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: No bootstrap required, opened a new log
I20260811 22:25:06.852312 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.839135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37645 (local address 127.0.0.1:35058)
0811 22:25:06.842797 (+ 3662us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.842811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.842934 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.843226 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.843229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.843403 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.843509 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.843516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.844521 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.844526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.845381 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.845395 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.846063 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.846067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.846164 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.846678 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.846693 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.848279 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.850531 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.850535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.850538 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.850808 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.851136 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.851139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.851141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.851233 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.851764 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.851767 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.851877 (+ 110us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.851972 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.852146 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3583}
I20260811 22:25:06.852878 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.839237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35058 (local address 127.29.3.125:37645)
0811 22:25:06.839362 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.839366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.842868 (+ 3502us) server_negotiation.cc:407] Connection header received
0811 22:25:06.843055 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.843058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.843110 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.843184 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.843630 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.844366 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.845530 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.845852 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.845939 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.848479 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.848483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.848485 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.848525 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.850246 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.850927 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.850931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.850932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.851019 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.851332 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.851335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.851337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.851610 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.851715 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.852606 (+ 891us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.852740 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:25:06.852741 4639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.852978 4639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512f3ce6f7594218b766c7514c1426bd, State: Initialized, Role: FOLLOWER
I20260811 22:25:06.853330 4639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.853570 4901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:06.854691 4781 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.855012 4781 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8351e8b72634b609d2eaf63e8b542bd, State: Initialized, Role: FOLLOWER
I20260811 22:25:06.855393 4781 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.855463 4639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: configured and running, proceeding with master startup.
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:06.856945 4781 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: configured and running, proceeding with master startup.
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:25:06.861478 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.842634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.29.3.126:33197)
0811 22:25:06.845711 (+ 3077us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.845715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.845729 (+ 14us) server_negotiation.cc:407] Connection header received
0811 22:25:06.845767 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.845770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.845812 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.845893 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.850461 (+ 4568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.851214 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.852674 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.853017 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.853097 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.856764 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.856770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.856772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.856804 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.858800 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.859464 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.859469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.859471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.859567 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.860164 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.860168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.860170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.860623 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.860797 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.861191 (+ 394us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.861323 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2998}
I20260811 22:25:06.862989 4910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.854119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:36633 (local address 127.0.0.1:46452)
0811 22:25:06.854240 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:06.854253 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:06.854369 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:06.854734 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:06.854737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:06.854909 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:06.855015 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.855022 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.855967 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.855970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.856741 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:06.856754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.857439 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.857442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:06.857515 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.857931 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:06.857946 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:06.859509 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:06.861405 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.861407 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.861409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.861630 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.861885 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:06.861887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:06.861889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:06.861962 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:06.862415 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:06.862417 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:06.862498 (+ 81us) client_negotiation.cc:769] Sending connection context
0811 22:25:06.862591 (+ 93us) client_negotiation.cc:240] Negotiation successful
0811 22:25:06.862757 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:25:06.863516 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.854133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.29.3.124:36633)
0811 22:25:06.854241 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.854245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.854436 (+ 191us) server_negotiation.cc:407] Connection header received
0811 22:25:06.854510 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.854514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.854569 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.854634 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.855142 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.855856 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.856862 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.857222 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.857319 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.859637 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.859641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.859644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.859669 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.861294 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.861726 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.861729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.861730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.861805 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.862041 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.862044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.862045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.862268 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.862383 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.863249 (+ 866us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.863378 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 22:25:06.865288 4945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:25:06.865378 4945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:25:06.867266 4901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: Bootstrap starting.
Aug 11 22:25:06 dist-test-slave-4tdj krb5kdc[4498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:25:06.870342 4901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:06.866829 4526 master_runner.cc:186] Error getting master registration for 127.29.3.124:36633: OK, instance_id { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" instance_seqno: 1786487106809608 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:25:06.871532 4526 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:25:06.874231 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.863433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.29.3.125:37645)
0811 22:25:06.863546 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.863550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.864043 (+ 493us) server_negotiation.cc:407] Connection header received
0811 22:25:06.864084 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.864087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.864132 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.864194 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.865040 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.865873 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.866871 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.867192 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.867288 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.870429 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.870467 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.870472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.870506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.872225 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.872734 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.872738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.872740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.872837 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.873146 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.873150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.873151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.873393 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.873561 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.873948 (+ 387us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.874080 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260811 22:25:06.875571 4969 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:25:06.875665 4969 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:25:06.872632 4650 master_runner.cc:186] Error getting master registration for 127.29.3.124:36633: OK, instance_id { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" instance_seqno: 1786487106809608 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:25:06.875931 4918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.876008 4650 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:25:06.876061 4918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.876355 4916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.876490 4916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.877506 4901 log.cc:824] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:06.878500 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:06.879760 4599 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "512f3ce6f7594218b766c7514c1426bd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48122
I20260811 22:25:06.880043 4599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 22:25:06.880170 4599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:06.881593 4599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.883492 4732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "512f3ce6f7594218b766c7514c1426bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b8351e8b72634b609d2eaf63e8b542bd"
I20260811 22:25:06.883754 4732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:06.883977 4901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: No bootstrap required, opened a new log
I20260811 22:25:06.885142 4732 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512f3ce6f7594218b766c7514c1426bd in term 1.
I20260811 22:25:06.885519 4599 leader_election.cc:288] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [CANDIDATE]: Term 1 election: Requested vote from peers b8351e8b72634b609d2eaf63e8b542bd (127.29.3.125:37645), f9623c23e62f4b30a2aaa09c8b077f9c (127.29.3.124:36633)
I20260811 22:25:06.885636 4559 leader_election.cc:302] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512f3ce6f7594218b766c7514c1426bd, b8351e8b72634b609d2eaf63e8b542bd; no voters:
I20260811 22:25:06.886113 4916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:06.886621 4916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [term 1 LEADER]: Becoming Leader. State: Replica: 512f3ce6f7594218b766c7514c1426bd, State: Running, Role: LEADER
I20260811 22:25:06.887333 4916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.891036 4916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512f3ce6f7594218b766c7514c1426bd. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.891379 4916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:06.899608 4644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.887915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.29.3.126:33197)
0811 22:25:06.888032 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.888037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.890136 (+ 2099us) server_negotiation.cc:407] Connection header received
0811 22:25:06.890372 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.890376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.890426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.890475 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.891490 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.892278 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.893464 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.893810 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.893934 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.895778 (+ 1844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.895782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.895785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.895818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.897540 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.898069 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.898073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.898075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.898185 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.898547 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.898551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.898553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.898789 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.898916 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.899296 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.899436 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 22:25:06.899725 4901 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.900075 4901 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9623c23e62f4b30a2aaa09c8b077f9c, State: Initialized, Role: FOLLOWER
I20260811 22:25:06.900502 4901 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } }
I20260811 22:25:06.901940 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.902372 4858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "512f3ce6f7594218b766c7514c1426bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c"
I20260811 22:25:06.902667 4858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:06.904227 4858 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512f3ce6f7594218b766c7514c1426bd in term 1.
I20260811 22:25:06.902278 4901 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:06.908982 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.888671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.29.3.125:37645)
0811 22:25:06.888918 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.888922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.894545 (+ 5623us) server_negotiation.cc:407] Connection header received
0811 22:25:06.894675 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.894678 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.894770 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.894830 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.895633 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.896436 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.897578 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.897910 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.898008 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.904163 (+ 6155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.904168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.904171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.904204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.906116 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.907527 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.907531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.907532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.907630 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.907950 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.907954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.907956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.908231 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.908343 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.908686 (+ 343us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.908820 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260811 22:25:06.910622 4999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:06.902249 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.915432 4999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260811 22:25:06.915506 5022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:25:06.915650 5022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:25:06.917304 4913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:06.889629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46456 (local address 127.29.3.124:36633)
0811 22:25:06.889768 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:06.889773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:06.900553 (+ 10780us) server_negotiation.cc:407] Connection header received
0811 22:25:06.900748 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:06.900752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:06.900807 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:06.900858 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:06.901739 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.902469 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.903608 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:06.903944 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:06.904024 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:06.907490 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:06.907494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:06.907497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:06.907527 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:06.911886 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.915785 (+ 3899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.915789 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.915791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.915895 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:06.916233 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:06.916237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:06.916238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:06.916487 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:06.916624 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:06.917005 (+ 381us) server_negotiation.cc:299] Negotiation successful
0811 22:25:06.917145 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:25:06.929672 4858 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [term 1 FOLLOWER]: Refusing update from remote peer 512f3ce6f7594218b766c7514c1426bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:06.929821 4732 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [term 1 FOLLOWER]: Refusing update from remote peer 512f3ce6f7594218b766c7514c1426bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:06.930533 4977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:06.930770 4916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:06.939625 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512f3ce6f7594218b766c7514c1426bd. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.939806 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.940863 4918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512f3ce6f7594218b766c7514c1426bd. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.941021 4918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.951943 4916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.952100 4916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:06.953351 4918 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.953504 4918 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.958837 4977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.958985 4977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:06.960454 5000 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "512f3ce6f7594218b766c7514c1426bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512f3ce6f7594218b766c7514c1426bd" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 33197 } } peers { permanent_uuid: "b8351e8b72634b609d2eaf63e8b542bd" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37645 } } peers { permanent_uuid: "f9623c23e62f4b30a2aaa09c8b077f9c" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 36633 } } }
I20260811 22:25:06.960585 5000 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:06.963459 4999 catalog_manager.cc:1347] Generated new cluster ID: c040efe455db4e5cb84fc017c8ea66fc
I20260811 22:25:06.963533 4999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:06.964864 5030 mvcc.cc:205] Tried to move back new op lower bound from 7317451189973958656 to 7317451189818073088. Current Snapshot: MvccSnapshot[applied={T|T < 7317451189973958656}]
I20260811 22:25:06.965783 5034 mvcc.cc:205] Tried to move back new op lower bound from 7317451189973958656 to 7317451189818073088. Current Snapshot: MvccSnapshot[applied={T|T < 7317451189973958656}]
I20260811 22:25:06.983943 4999 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:06.985350 4999 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:07.006839 4999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 512f3ce6f7594218b766c7514c1426bd: Generated new TSK 0
I20260811 22:25:07.007774 4999 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:25:07.060674 4644 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.057326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48148 (local address 127.29.3.126:33197)
0811 22:25:07.057458 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.057463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.059442 (+ 1979us) server_negotiation.cc:407] Connection header received
0811 22:25:07.059666 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.059670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.059733 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.059786 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.060533 (+ 747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48148: BlockingRecv error: recv got EOF from 127.0.0.1:48148 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 22:25:07.062361 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.058113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.29.3.125:37645)
0811 22:25:07.058230 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.058235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.061364 (+ 3129us) server_negotiation.cc:407] Connection header received
0811 22:25:07.061549 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.061553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.061606 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.061655 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.062184 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35082: BlockingRecv error: recv got EOF from 127.0.0.1:35082 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260811 22:25:07.063954 4913 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.058833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.29.3.124:36633)
0811 22:25:07.058940 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.058944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.063020 (+ 4076us) server_negotiation.cc:407] Connection header received
0811 22:25:07.063197 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.063200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.063253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.063300 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.063823 (+ 523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46458: BlockingRecv error: recv got EOF from 127.0.0.1:46458 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260811 22:25:07.069171 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.067359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.29.3.125:37645)
0811 22:25:07.067478 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.067482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.068030 (+ 548us) server_negotiation.cc:407] Connection header received
0811 22:25:07.068278 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.068283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.068374 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.068429 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.069043 (+ 614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35084: BlockingRecv error: recv got EOF from 127.0.0.1:35084 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260811 22:25:07.074875 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.073217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.29.3.125:37645)
0811 22:25:07.073332 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.073336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.073904 (+ 568us) server_negotiation.cc:407] Connection header received
0811 22:25:07.074092 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.074096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.074151 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.074200 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.074760 (+ 560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35086: BlockingRecv error: recv got EOF from 127.0.0.1:35086 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260811 22:25:07.080804 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.079126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.29.3.125:37645)
0811 22:25:07.079235 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.079239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.079765 (+ 526us) server_negotiation.cc:407] Connection header received
0811 22:25:07.079985 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.079989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.080053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.080110 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.080678 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35092: BlockingRecv error: recv got EOF from 127.0.0.1:35092 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260811 22:25:07.088326 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.087116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.29.3.125:37645)
0811 22:25:07.087301 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.087306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.087498 (+ 192us) server_negotiation.cc:407] Connection header received
0811 22:25:07.087631 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.087635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.087701 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.087782 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.088225 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35108: BlockingRecv error: recv got EOF from 127.0.0.1:35108 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260811 22:25:07.100163 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.098902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.29.3.125:37645)
0811 22:25:07.099045 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.099050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.099268 (+ 218us) server_negotiation.cc:407] Connection header received
0811 22:25:07.099419 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.099422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.099474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.099544 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.100051 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35114: BlockingRecv error: recv got EOF from 127.0.0.1:35114 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 22:25:07.119740 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.118534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35126 (local address 127.29.3.125:37645)
0811 22:25:07.118689 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.118692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.118860 (+ 168us) server_negotiation.cc:407] Connection header received
0811 22:25:07.119007 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.119010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.119060 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.119131 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.119585 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35126: BlockingRecv error: recv got EOF from 127.0.0.1:35126 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 22:25:07.155506 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.154194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.29.3.125:37645)
0811 22:25:07.154381 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.154385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.154537 (+ 152us) server_negotiation.cc:407] Connection header received
0811 22:25:07.154725 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.154728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.154781 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.154875 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.155375 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35136: BlockingRecv error: recv got EOF from 127.0.0.1:35136 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260811 22:25:07.223130 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.221870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.29.3.125:37645)
0811 22:25:07.222066 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.222071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.222171 (+ 100us) server_negotiation.cc:407] Connection header received
0811 22:25:07.222361 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.222363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.222412 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.222484 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.223000 (+ 516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35152: BlockingRecv error: recv got EOF from 127.0.0.1:35152 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260811 22:25:07.354355 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.353086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.29.3.125:37645)
0811 22:25:07.353223 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.353227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.353362 (+ 135us) server_negotiation.cc:407] Connection header received
0811 22:25:07.353493 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.353496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.353545 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.353636 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.354186 (+ 550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35158: BlockingRecv error: recv got EOF from 127.0.0.1:35158 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 22:25:07.613626 4791 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:07.612536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.29.3.125:37645)
0811 22:25:07.612710 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:07.612715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:07.612731 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:25:07.612877 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:07.612879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:07.612930 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:07.612998 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:07.613499 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35170: BlockingRecv error: recv got EOF from 127.0.0.1:35170 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:25:07.877130 4969 catalog_manager.cc:1259] Loaded cluster ID: c040efe455db4e5cb84fc017c8ea66fc
I20260811 22:25:07.877244 4969 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: loading cluster ID for follower catalog manager: success
I20260811 22:25:07.879072 4969 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: acquiring CA information for follower catalog manager: success
I20260811 22:25:07.880353 4969 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b8351e8b72634b609d2eaf63e8b542bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:25:07.917135 5022 catalog_manager.cc:1259] Loaded cluster ID: c040efe455db4e5cb84fc017c8ea66fc
I20260811 22:25:07.917233 5022 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: loading cluster ID for follower catalog manager: success
I20260811 22:25:07.919140 5022 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: acquiring CA information for follower catalog manager: success
I20260811 22:25:07.920396 5022 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f9623c23e62f4b30a2aaa09c8b077f9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:25:08.131851 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:08.127872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.29.3.125:37645)
0811 22:25:08.128130 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:08.128134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:08.128149 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:25:08.128271 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:08.128273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:08.128411 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:08.128514 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:08.128758 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.129294 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.130340 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.130586 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.130678 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:08.130967 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:08.131443 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:08.131540 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:08.131617 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 22:25:08.131679 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
I20260811 22:25:08.139492 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:08.135446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.29.3.124:36633)
0811 22:25:08.135717 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:08.135721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:08.135738 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:25:08.135840 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:08.135843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:08.135969 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:08.136082 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:08.136320 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.136887 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.137820 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.138058 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.138151 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:08.138530 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:08.139064 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:08.139188 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:08.139260 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 22:25:08.139325 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":68,"threads_started":1}
I20260811 22:25:08.147050 5169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:08.143037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.29.3.125:37645)
0811 22:25:08.143208 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:08.143212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:08.143479 (+ 267us) server_negotiation.cc:407] Connection header received
0811 22:25:08.143569 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:08.143573 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:08.143627 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:08.143687 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:08.144037 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.144818 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.145751 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.146064 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.146163 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:08.146548 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:08.146652 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:08.146730 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:08.146821 (+ 91us) server_negotiation.cc:299] Negotiation successful
0811 22:25:08.146891 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 22:25:08.147423 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:08.142951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.29.3.124:36633)
0811 22:25:08.143065 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:08.143068 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:08.143153 (+ 85us) server_negotiation.cc:407] Connection header received
0811 22:25:08.143378 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:08.143381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:08.143438 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:08.143502 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:08.143981 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.144813 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.145947 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.146212 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.146340 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:08.146675 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:08.146756 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:08.146816 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:08.147114 (+ 298us) server_negotiation.cc:299] Negotiation successful
0811 22:25:08.147293 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:25:08.148242 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:08.142691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.29.3.126:33197)
0811 22:25:08.142942 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:08.142947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:08.143117 (+ 170us) server_negotiation.cc:407] Connection header received
0811 22:25:08.143496 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:08.143500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:08.143685 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:08.143809 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:08.144136 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.144668 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.145854 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:08.146184 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:08.146280 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:08.146692 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:08.147242 (+ 550us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:08.147335 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:08.148042 (+ 707us) server_negotiation.cc:299] Negotiation successful
0811 22:25:08.148103 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260811 22:25:08.151599 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4526
I20260811 22:25:08.172189 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4650
I20260811 22:25:08.200551 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 4794
2026-08-11T22:25:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3361 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 22:25:08 dist-test-slave-4tdj krb5kdc[5209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 22:25:08 dist-test-slave-4tdj krb5kdc[5209](info): set up 2 sockets
Aug 11 22:25:08 dist-test-slave-4tdj krb5kdc[5209](info): commencing operation
krb5kdc: starting...
W20260811 22:25:09.413203 29709 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/test-user.keytab.
Aug 11 22:25:09 dist-test-slave-4tdj krb5kdc[5209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T22:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T22:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.3.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
Entry for principal kudu/127.29.3.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab.
I20260811 22:25:09.547107 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:45707
--webserver_interface=127.29.3.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:45707,127.29.3.125:37199,127.29.3.124:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 22:25:09.664155 5227 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:25:09.664492 5227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:09.664517 5227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:09.664567 5227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:09.668792 5227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:09.668874 5227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:09.668893 5227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:09.668910 5227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:09.668924 5227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:09.674557 5227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:45707,127.29.3.125:37199,127.29.3.124:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.126.keytab
--principal=kudu/127.29.3.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.126:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.3.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:09.675839 5227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:09.676934 5227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:09.685741 5236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:09.685731 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:09.685830 5227 server_base.cc:1034] running on GCE node
W20260811 22:25:09.685732 5238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:09.686753 5227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:09.687937 5227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:09.689103 5227 hybrid_clock.cc:648] HybridClock initialized: now 1786487109689085 us; error 31 us; skew 500 ppm
Aug 11 22:25:09 dist-test-slave-4tdj krb5kdc[5209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:09.691996 5227 init.cc:377] Logged in from keytab as kudu/127.29.3.126@KRBTEST.COM (short username kudu)
I20260811 22:25:09.692996 5227 webserver.cc:460] Webserver started at http://127.29.3.126:32859/ using document root <none> and password file <none>
I20260811 22:25:09.693576 5227 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:09.693651 5227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:09.693893 5227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:09.696133 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data/instance:
uuid: "10f468027dda432ea0e552fb89c12e61"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-4tdj"
server_key: "2cb244520eae2667d569baccb1795df2"
server_key_iv: "8c22887765c394d487137c8e7e6718a8"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.696743 5227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal/instance:
uuid: "10f468027dda432ea0e552fb89c12e61"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-4tdj"
server_key: "2cb244520eae2667d569baccb1795df2"
server_key_iv: "8c22887765c394d487137c8e7e6718a8"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.700825 5227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 22:25:09.703576 5246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:09.704941 5227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 22:25:09.705075 5227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal
uuid: "10f468027dda432ea0e552fb89c12e61"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-4tdj"
server_key: "2cb244520eae2667d569baccb1795df2"
server_key_iv: "8c22887765c394d487137c8e7e6718a8"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.705188 5227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:09.738041 5227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:09.741449 5227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:09.741662 5227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:09.749186 5227 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.126:45707
I20260811 22:25:09.749456 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.126:45707 every 8 connection(s)
I20260811 22:25:09.750216 5227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/data/info.pb
I20260811 22:25:09.753149 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:09.754971 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5227
I20260811 22:25:09.755062 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-0/wal/instance
I20260811 22:25:09.755337 29709 external_mini_cluster.cc:1428] Setting key 06986e7824840c4dff4390e69b5377d8
I20260811 22:25:09.757184 5339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.3.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.125@KRBTEST.COM" created.
Aug 11 22:25:09 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:09.772596 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.759583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.29.3.126:45707)
0811 22:25:09.760022 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:09.760041 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:09.760304 (+ 263us) server_negotiation.cc:407] Connection header received
0811 22:25:09.761376 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:09.761380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:09.761574 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:09.761737 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:09.762662 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.763551 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.764554 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.764973 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.765132 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.767485 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:09.767490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:09.767501 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:09.767523 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:09.769345 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.769859 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.769869 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.769871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:09.770002 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.770376 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.770379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.770381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:09.770813 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:09.770923 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:09.771431 (+ 508us) server_negotiation.cc:299] Negotiation successful
0811 22:25:09.771740 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":289,"thread_start_us":120,"threads_started":1}
I20260811 22:25:09.772598 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.759583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:45707 (local address 127.0.0.1:43298)
0811 22:25:09.760160 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.760199 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:09.761072 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:09.761765 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:09.761771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:09.762212 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:09.762520 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.762536 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.763677 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.763680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.764443 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.764454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.765125 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.765128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.765253 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.765809 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:09.765827 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:09.767348 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:09.769483 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.769492 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.769501 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.769745 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:09.770093 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.770097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.770098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.770259 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:09.770932 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:09.770939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:09.771127 (+ 188us) client_negotiation.cc:769] Sending connection context
0811 22:25:09.771359 (+ 232us) client_negotiation.cc:240] Negotiation successful
0811 22:25:09.771833 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":140,"thread_start_us":158,"threads_started":1}
I20260811 22:25:09.775331 5339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:09.776117 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.775835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33532)
0811 22:25:09.775963 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.776043 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 22:25:09.776405 5252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
I20260811 22:25:09.777405 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.777171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33534)
0811 22:25:09.777284 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.777344 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 22:25:09.777585 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:37199: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab.
Entry for principal kudu/127.29.3.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab.
I20260811 22:25:09.785754 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:37199
--webserver_interface=127.29.3.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab
--principal=kudu/127.29.3.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:45707,127.29.3.125:37199,127.29.3.124:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:25:09.826864 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } attempt: 1
I20260811 22:25:09.827831 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.827451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33544)
0811 22:25:09.827641 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.827734 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260811 22:25:09.828594 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.828364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33550)
0811 22:25:09.828470 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.828527 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 22:25:09.828775 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:37199: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
I20260811 22:25:09.896958 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } attempt: 2
I20260811 22:25:09.897955 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.897590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33554)
0811 22:25:09.897773 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.897854 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 22:25:09.898772 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.898525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33568)
0811 22:25:09.898638 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.898698 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 22:25:09.898937 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.125:37199: Network error: Client connection negotiation failed: client connection to 127.29.3.125:37199: connect: Connection refused (error 111)
W20260811 22:25:09.905566 5360 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:25:09.905901 5360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:09.905927 5360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:09.905972 5360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:09.910149 5360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:09.910228 5360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:09.910246 5360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:09.910262 5360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:09.910276 5360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:09.915997 5360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:45707,127.29.3.125:37199,127.29.3.124:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.125.keytab
--principal=kudu/127.29.3.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.125:37199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.3.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:09.917114 5360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:09.918116 5360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 22:25:09.927095 5360 server_base.cc:1034] running on GCE node
W20260811 22:25:09.927086 5369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:09.927049 5371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:09.927055 5374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:09.927970 5360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:09.929176 5360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:09.930359 5360 hybrid_clock.cc:648] HybridClock initialized: now 1786487109930333 us; error 39 us; skew 500 ppm
Aug 11 22:25:09 dist-test-slave-4tdj krb5kdc[5209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:09.933295 5360 init.cc:377] Logged in from keytab as kudu/127.29.3.125@KRBTEST.COM (short username kudu)
I20260811 22:25:09.934268 5360 webserver.cc:460] Webserver started at http://127.29.3.125:43495/ using document root <none> and password file <none>
I20260811 22:25:09.934877 5360 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:09.934942 5360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:09.935150 5360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:09.937470 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data/instance:
uuid: "b26a875b11664be58f4a80519334228f"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-4tdj"
server_key: "c1a9f758ab158923c0b6124044d67a50"
server_key_iv: "9c9dacc1724ef6204c5413746ed22746"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.938057 5360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal/instance:
uuid: "b26a875b11664be58f4a80519334228f"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-4tdj"
server_key: "c1a9f758ab158923c0b6124044d67a50"
server_key_iv: "9c9dacc1724ef6204c5413746ed22746"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.942198 5360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 22:25:09.945073 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:09.946575 5360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:09.946728 5360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal
uuid: "b26a875b11664be58f4a80519334228f"
format_stamp: "Formatted at 2026-08-11 22:25:09 on dist-test-slave-4tdj"
server_key: "c1a9f758ab158923c0b6124044d67a50"
server_key_iv: "9c9dacc1724ef6204c5413746ed22746"
server_key_version: "encryptionkey@0"
I20260811 22:25:09.946852 5360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:09.960345 5360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:09.963625 5360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:09.963836 5360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:09.971053 5360 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.125:37199
I20260811 22:25:09.971347 5472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.125:37199 every 8 connection(s)
I20260811 22:25:09.972064 5360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/data/info.pb
I20260811 22:25:09.973557 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5360
I20260811 22:25:09.973680 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-1/wal/instance
I20260811 22:25:09.974010 29709 external_mini_cluster.cc:1428] Setting key eb83dd72813fa309ea9c386a6efc507a
I20260811 22:25:09.975689 5473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:09.980027 5473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.3.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.3.124@KRBTEST.COM" created.
I20260811 22:25:09.990097 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } attempt: 3
Aug 11 22:25:09 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:09.995759 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.982092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.29.3.126:45707)
0811 22:25:09.982259 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:09.982265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:09.983202 (+ 937us) server_negotiation.cc:407] Connection header received
0811 22:25:09.984261 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:09.984265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:09.984336 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:09.984412 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:09.985717 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.986252 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.987617 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.987875 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.987972 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.991058 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:09.991063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:09.991066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:09.991098 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:09.993350 (+ 2252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.993934 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.993938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.993939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:09.994020 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.994359 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.994363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.994365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:09.994587 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:09.994686 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:09.995333 (+ 647us) server_negotiation.cc:299] Negotiation successful
0811 22:25:09.995453 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 22:25:09.996398 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.982511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:45707 (local address 127.0.0.1:43314)
0811 22:25:09.983038 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.983087 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:09.984029 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:09.984620 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:09.984628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:09.985001 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:09.985559 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.985575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.986405 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.986407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.987466 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.987479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.988043 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.988047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.988266 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.988956 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:09.988975 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:09.990838 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:09.993464 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.993471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.993489 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.993847 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:09.994102 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.994105 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.994107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.994248 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:09.994696 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:09.994704 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:09.995046 (+ 342us) client_negotiation.cc:769] Sending connection context
0811 22:25:09.995228 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 22:25:09.995490 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
Aug 11 22:25:09 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:25:09.997929 5473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:10.001312 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.990718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.29.3.125:37199)
0811 22:25:09.991187 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:09.991197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:09.991217 (+ 20us) server_negotiation.cc:407] Connection header received
0811 22:25:09.991253 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:09.991258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:09.991457 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:09.991597 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:09.992040 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.992893 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.993771 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.994384 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.994500 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.997690 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:09.997700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:09.997714 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:09.997741 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:09.999582 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:09.999985 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:09.999991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:09.999993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.000068 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.000375 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.000379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.000380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.000619 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.000726 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.001033 (+ 307us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.001159 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":162,"threads_started":1}
I20260811 22:25:10.001333 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.990609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33572)
0811 22:25:09.990828 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.990843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:09.990931 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:09.991598 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:09.991601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:09.991792 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:09.991929 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.991937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.993005 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.993009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.993642 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.993653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:09.994503 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:09.994507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:09.994610 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:09.994999 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:09.995011 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:09.997570 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:09.999683 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:09.999686 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:09.999689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:09.999882 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.000156 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.000159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.000161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.000264 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.000725 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.000728 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.000838 (+ 110us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.000942 (+ 104us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.001096 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:25:10.003867 5339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.3.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab.
Entry for principal kudu/127.29.3.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab.
I20260811 22:25:10.004733 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.004397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60320)
0811 22:25:10.004569 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.004621 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260811 22:25:10.005483 29709 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
/tmp/dist-test-taskCdqKk8/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:45371
--webserver_interface=127.29.3.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab
--principal=kudu/127.29.3.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.3.84:37377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.3.126:45707,127.29.3.125:37199,127.29.3.124:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 22:25:10.005697 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.005208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60328)
0811 22:25:10.005560 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.005617 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288}
W20260811 22:25:10.005846 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.007767 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.998486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33586)
0811 22:25:09.998594 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:09.998608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:09.998725 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:09.999262 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:09.999265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:09.999460 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:09.999557 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:09.999563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.000350 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.000353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.001033 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.001043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.002286 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.002316 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.002419 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.002886 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.002898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.004251 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.006258 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.006261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.006263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.006493 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.006760 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.006762 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.006763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.006831 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.007209 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.007211 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.007309 (+ 98us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.007418 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.007573 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 22:25:10.007791 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:09.998527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.29.3.125:37199)
0811 22:25:09.999065 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:09.999072 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:09.999084 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:25:09.999111 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:09.999113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:09.999146 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:09.999228 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:09.999658 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.000261 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.001146 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.001696 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.001840 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.004417 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.004421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.004423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.004443 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.006151 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.006589 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.006592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.006594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.006686 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.006900 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.006902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.006904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.007130 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.007223 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.007478 (+ 255us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.007635 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":474,"thread_start_us":125,"threads_started":1}
I20260811 22:25:10.008569 5473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
I20260811 22:25:10.009318 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.009032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60336)
0811 22:25:10.009151 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.009249 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 22:25:10.009538 5385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.010392 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.010181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60346)
0811 22:25:10.010260 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.010331 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260811 22:25:10.010543 5473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.036995 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 1
I20260811 22:25:10.037919 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.037619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60352)
0811 22:25:10.037768 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.037836 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 22:25:10.038810 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.038568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60366)
0811 22:25:10.038689 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.038740 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260811 22:25:10.039024 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.059810 5473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 1
I20260811 22:25:10.060786 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.060447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60372)
0811 22:25:10.060619 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.060694 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260811 22:25:10.061529 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.061322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60386)
0811 22:25:10.061416 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.061464 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 22:25:10.061667 5473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.114187 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 2
I20260811 22:25:10.115032 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.114731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60402)
0811 22:25:10.114885 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.114950 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 22:25:10.115923 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.115662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60412)
0811 22:25:10.115783 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.115845 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 22:25:10.116113 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.129827 5473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 2
I20260811 22:25:10.130685 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.130404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60428)
0811 22:25:10.130538 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.130606 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 22:25:10.131436 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.131241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60438)
0811 22:25:10.131331 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.131377 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260811 22:25:10.131556 5473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
W20260811 22:25:10.137796 5495 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 22:25:10.138098 5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 22:25:10.138127 5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 22:25:10.138166 5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 22:25:10.142273 5495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 22:25:10.142400 5495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 22:25:10.142418 5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 22:25:10.142436 5495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 22:25:10.142457 5495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 22:25:10.148092 5495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.3.84:37377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.3.126:45707,127.29.3.125:37199,127.29.3.124:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/krb5kdc/kudu_127.29.3.124.keytab
--principal=kudu/127.29.3.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.3.124:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.3.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 21:22:59 UTC on ccycloud.quasar-ckgmnv.root.comops.site
build id 4581473
I20260811 22:25:10.149228 5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 22:25:10.150388 5495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 22:25:10.159417 5510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 22:25:10.159479 5509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:10.159457 5495 server_base.cc:1034] running on GCE node
W20260811 22:25:10.159417 5512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 22:25:10.160354 5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 22:25:10.161576 5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 22:25:10.162752 5495 hybrid_clock.cc:648] HybridClock initialized: now 1786487110162732 us; error 34 us; skew 500 ppm
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487110, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 22:25:10.165699 5495 init.cc:377] Logged in from keytab as kudu/127.29.3.124@KRBTEST.COM (short username kudu)
I20260811 22:25:10.166850 5495 webserver.cc:460] Webserver started at http://127.29.3.124:34493/ using document root <none> and password file <none>
I20260811 22:25:10.167424 5495 fs_manager.cc:359] Metadata directory not provided
I20260811 22:25:10.167495 5495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 22:25:10.167727 5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 22:25:10.169912 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data/instance:
uuid: "aab804bcf9e04ee790a4da42d6eb6e75"
format_stamp: "Formatted at 2026-08-11 22:25:10 on dist-test-slave-4tdj"
server_key: "a003f24f7dc2c8e98a77fdce9079c6ea"
server_key_iv: "3e250288691a5c4191344c7b1d569444"
server_key_version: "encryptionkey@0"
I20260811 22:25:10.170548 5495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal/instance:
uuid: "aab804bcf9e04ee790a4da42d6eb6e75"
format_stamp: "Formatted at 2026-08-11 22:25:10 on dist-test-slave-4tdj"
server_key: "a003f24f7dc2c8e98a77fdce9079c6ea"
server_key_iv: "3e250288691a5c4191344c7b1d569444"
server_key_version: "encryptionkey@0"
I20260811 22:25:10.174547 5495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 22:25:10.177201 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 22:25:10.178611 5495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 22:25:10.178742 5495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data,/tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal
uuid: "aab804bcf9e04ee790a4da42d6eb6e75"
format_stamp: "Formatted at 2026-08-11 22:25:10 on dist-test-slave-4tdj"
server_key: "a003f24f7dc2c8e98a77fdce9079c6ea"
server_key_iv: "3e250288691a5c4191344c7b1d569444"
server_key_version: "encryptionkey@0"
I20260811 22:25:10.178855 5495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 22:25:10.215271 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 3
I20260811 22:25:10.216080 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.215785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60440)
0811 22:25:10.215932 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.215996 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 22:25:10.216790 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.216553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60442)
0811 22:25:10.216639 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.216686 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260811 22:25:10.216940 5339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.222705 5473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 3
I20260811 22:25:10.223289 5495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 22:25:10.223547 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.223227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60458)
0811 22:25:10.223393 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.223459 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 22:25:10.224262 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.224005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60472)
0811 22:25:10.224150 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.224202 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260811 22:25:10.224409 5473 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.3.124:45371: Network error: Client connection negotiation failed: client connection to 127.29.3.124:45371: connect: Connection refused (error 111)
I20260811 22:25:10.226789 5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 22:25:10.226979 5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 22:25:10.234149 5495 rpc_server.cc:307] RPC server started. Bound to: 127.29.3.124:45371
I20260811 22:25:10.234447 5601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.3.124:45371 every 8 connection(s)
I20260811 22:25:10.235209 5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/data/info.pb
I20260811 22:25:10.237308 29709 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu as pid 5495
I20260811 22:25:10.237414 29709 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskCdqKk8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786486810933177-29709-0/minicluster-data/master-2/wal/instance
I20260811 22:25:10.237702 29709 external_mini_cluster.cc:1428] Setting key 8a29d86557e8e2c3a05dd7e4ba53ecc0
I20260811 22:25:10.238349 5602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 22:25:10.242563 5602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487110, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:10.253557 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.239083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.29.3.124:45371)
0811 22:25:10.239578 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.239594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.239634 (+ 40us) server_negotiation.cc:407] Connection header received
0811 22:25:10.240330 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.240347 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.240702 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.241044 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.241872 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.242949 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.244176 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.244879 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.245040 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.247922 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.247927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.247937 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.247965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.249973 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.250489 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.250494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.250499 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.250612 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.250914 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.250917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.250919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.251481 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.251718 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.252167 (+ 449us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.252499 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":153,"threads_started":1}
I20260811 22:25:10.254141 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.244568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.29.3.126:45707)
0811 22:25:10.244698 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.244703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.245075 (+ 372us) server_negotiation.cc:407] Connection header received
0811 22:25:10.245303 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.245306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.245356 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.245439 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.246358 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.246923 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.248009 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.248233 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.248323 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.250812 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.250815 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.250817 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.250842 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.252317 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.252820 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.252824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.252826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.252912 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.253166 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.253170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.253171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.253413 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.253513 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.253903 (+ 390us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.254015 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 22:25:10.254670 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.244550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.126:45707 (local address 127.0.0.1:43318)
0811 22:25:10.244940 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.244997 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:10.245176 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:10.245511 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:10.245515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:10.246005 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:10.246188 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.246197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.247057 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.247060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.247898 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.247908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.248332 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.248336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.248449 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.248937 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.248953 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.250690 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.252452 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.252461 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.252466 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.252725 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.253006 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.253009 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.253010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.253090 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.253528 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.253534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.253660 (+ 126us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.254321 (+ 661us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.254524 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":138,"threads_started":1}
I20260811 22:25:10.255877 5602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487110, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.126@KRBTEST.COM
I20260811 22:25:10.264673 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.256427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.125:37199 (local address 127.0.0.1:33590)
0811 22:25:10.256586 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.256598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:10.256665 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:10.256886 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:10.256889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:10.257053 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:10.257193 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.257200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.258205 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.258207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.259080 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.259091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.259651 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.259655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.259758 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.260299 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.260313 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.261654 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.263294 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.263297 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.263299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.263487 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.263744 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.263748 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.263750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.263827 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.264180 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.264183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.264279 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.264362 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.264502 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260811 22:25:10.264707 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.256521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.29.3.125:37199)
0811 22:25:10.256638 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.256642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.256654 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:25:10.256750 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.256752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.256790 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.256853 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.257301 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.258119 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.259191 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.259547 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.259653 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.261799 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.261802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.261805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.261836 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.263189 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.263581 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.263584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.263586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.263662 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.263903 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.263906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.263908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.264096 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.264180 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.264422 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.264544 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 22:25:10.265501 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.256046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43324 (local address 127.29.3.126:45707)
0811 22:25:10.256157 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.256160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.256197 (+ 37us) server_negotiation.cc:407] Connection header received
0811 22:25:10.256342 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.256345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.256393 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.256462 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.257311 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.258095 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.259191 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.259547 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.259653 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.262380 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.262383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.262385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.262409 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.263768 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.264159 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.264162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.264163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.264252 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.264502 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.264505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.264507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.264678 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.264921 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.265219 (+ 298us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.265357 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:25:10.265712 5602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487110, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.124@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.3.125@KRBTEST.COM
I20260811 22:25:10.274015 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.266415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.29.3.124:45371)
0811 22:25:10.266539 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.266543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.266554 (+ 11us) server_negotiation.cc:407] Connection header received
0811 22:25:10.266581 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.266583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.266615 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.266682 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.267055 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.267626 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.268328 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.268563 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.268645 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.270809 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.270812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.270814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.270836 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.272243 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.272746 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.272750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.272752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.272832 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.273089 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.273092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.273094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.273327 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.273432 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.273719 (+ 287us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.273857 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":43}
I20260811 22:25:10.274021 5607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.266230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60492)
0811 22:25:10.266385 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.266397 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:10.266464 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:10.266779 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:10.266781 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:10.266907 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:10.266968 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.266972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.267717 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.267720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.268223 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.268231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.268653 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.268657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.268761 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.269175 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.269189 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.270648 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.272362 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.272365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.272367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.272648 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.272924 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.272928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.272929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.273007 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.273432 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.273434 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.273520 (+ 86us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.273650 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.273806 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":17}
I20260811 22:25:10.275190 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.266788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.29.3.125:37199)
0811 22:25:10.266896 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.266899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.266912 (+ 13us) server_negotiation.cc:407] Connection header received
0811 22:25:10.266989 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.266992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.267033 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.267103 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.267628 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.268125 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.268837 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.269135 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.269257 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.271985 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.271990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.271993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.272024 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.273565 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.274014 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.274017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.274019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.274106 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.274405 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.274408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.274410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.274628 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.274896 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.274967 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.275082 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 22:25:10.277927 5602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: Bootstrap starting.
I20260811 22:25:10.278656 29709 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 22:25:10.280689 5602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:10.279788 5295 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "10f468027dda432ea0e552fb89c12e61"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43324
I20260811 22:25:10.281639 5602 log.cc:824] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:10.283761 5602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: No bootstrap required, opened a new log
I20260811 22:25:10.287698 5602 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.287915 5602 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab804bcf9e04ee790a4da42d6eb6e75, State: Initialized, Role: FOLLOWER
I20260811 22:25:10.288233 5602 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.288743 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.288870 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.289685 5602 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 22:25:10.294967 5626 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:25:10.295054 5626 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:25:10.293756 5495 master_runner.cc:186] Error getting master registration for 127.29.3.126:45707: OK, instance_id { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" instance_seqno: 1786487109747155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:25:10.297160 5495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:25:10.367570 5473 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 4
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.125@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:25:10.376726 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.368285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.29.3.124:45371)
0811 22:25:10.368463 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.368467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.368484 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:25:10.368592 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.368595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.368647 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.368721 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.369376 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.369905 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.370745 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.370990 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.371083 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.373590 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.373594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.373596 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.373626 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.375023 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.375518 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.375522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.375524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.375608 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.375872 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.375876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.375878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.376079 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.376169 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.376447 (+ 278us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.376569 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 22:25:10.376726 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.368206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60506)
0811 22:25:10.368359 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.368377 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:10.368490 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:10.368766 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:10.368770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:10.368968 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:10.369200 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.369219 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.370024 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.370028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.370605 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.370618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.371105 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.371109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.371191 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.371797 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.371812 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.373419 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.375144 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.375147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.375150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.375416 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.375697 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.375701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.375703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.375790 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.376172 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.376176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.376288 (+ 112us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.376379 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.376538 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 22:25:10.379863 5473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: Bootstrap starting.
I20260811 22:25:10.381115 5339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } attempt: 4
I20260811 22:25:10.382943 5473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:10.383967 5473 log.cc:824] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: Log is configured to *not* fsync() on all Append() calls
I20260811 22:25:10.386364 5473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: No bootstrap required, opened a new log
Aug 11 22:25:10 dist-test-slave-4tdj krb5kdc[5209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786487109, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.3.126@KRBTEST.COM for kudu/127.29.3.124@KRBTEST.COM
I20260811 22:25:10.390493 5473 raft_consensus.cc:348] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.390796 5473 raft_consensus.cc:729] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b26a875b11664be58f4a80519334228f, State: Initialized, Role: FOLLOWER
I20260811 22:25:10.391173 5473 consensus_queue.cc:260] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.391341 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.381650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.3.124:45371 (local address 127.0.0.1:60514)
0811 22:25:10.382038 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0811 22:25:10.382054 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 22:25:10.382166 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 22:25:10.382473 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 22:25:10.382476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 22:25:10.382692 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0811 22:25:10.382828 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.382837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.383868 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.383872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.384747 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 22:25:10.384761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.385398 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.385401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 22:25:10.385494 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.386048 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 22:25:10.386062 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 22:25:10.387600 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 22:25:10.389577 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.389581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.389583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.389876 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.390186 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 22:25:10.390189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 22:25:10.390191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 22:25:10.390317 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 22:25:10.390763 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 22:25:10.390766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 22:25:10.390879 (+ 113us) client_negotiation.cc:769] Sending connection context
0811 22:25:10.390993 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 22:25:10.391175 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294}
I20260811 22:25:10.391351 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.381883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.29.3.124:45371)
0811 22:25:10.382028 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.382033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.382118 (+ 85us) server_negotiation.cc:407] Connection header received
0811 22:25:10.382255 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.382258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.382336 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.382437 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.382981 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.383745 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.384948 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.385275 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.385383 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.387797 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.387801 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.387805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.387839 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.389463 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.389983 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.389987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.389989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.390088 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.390401 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.390404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.390406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.390662 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.390775 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.391068 (+ 293us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.391201 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 22:25:10.392889 5473 sys_catalog.cc:564] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:10.394909 5339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61: Bootstrap starting.
I20260811 22:25:10.395802 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.396298 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.398097 5339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61: Neither blocks nor log segments found. Creating new log.
I20260811 22:25:10.399075 5339 log.cc:824] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61: Log is configured to *not* fsync() on all Append() calls
W20260811 22:25:10.399318 5652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 22:25:10.399428 5652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 22:25:10.397141 5360 master_runner.cc:186] Error getting master registration for 127.29.3.126:45707: OK, instance_id { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" instance_seqno: 1786487109747155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 22:25:10.400382 5360 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:25:10.401530 5339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61: No bootstrap required, opened a new log
I20260811 22:25:10.404745 5339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.404997 5339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f468027dda432ea0e552fb89c12e61, State: Initialized, Role: FOLLOWER
I20260811 22:25:10.405359 5339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.406047 5658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.406260 5658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.406952 5339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 22:25:10.407042 5295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 22:25:10.407159 5295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:10.408550 5295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.409690 5295 leader_election.cc:288] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [CANDIDATE]: Term 1 election: Requested vote from peers b26a875b11664be58f4a80519334228f (127.29.3.125:37199), aab804bcf9e04ee790a4da42d6eb6e75 (127.29.3.124:45371)
I20260811 22:25:10.410710 5429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10f468027dda432ea0e552fb89c12e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b26a875b11664be58f4a80519334228f"
I20260811 22:25:10.410986 5429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:10.412433 5429 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f468027dda432ea0e552fb89c12e61 in term 1.
I20260811 22:25:10.412688 5559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10f468027dda432ea0e552fb89c12e61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aab804bcf9e04ee790a4da42d6eb6e75"
I20260811 22:25:10.413017 5559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [term 0 FOLLOWER]: Advancing to term 1
I20260811 22:25:10.414526 5559 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10f468027dda432ea0e552fb89c12e61 in term 1.
I20260811 22:25:10.415098 5249 leader_election.cc:302] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 10f468027dda432ea0e552fb89c12e61, aab804bcf9e04ee790a4da42d6eb6e75, b26a875b11664be58f4a80519334228f; no voters:
I20260811 22:25:10.422319 5658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 22:25:10.422883 5658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [term 1 LEADER]: Becoming Leader. State: Replica: 10f468027dda432ea0e552fb89c12e61, State: Running, Role: LEADER
I20260811 22:25:10.423189 5227 master_runner.cc:263] No leader master found from master 10f468027dda432ea0e552fb89c12e61
I20260811 22:25:10.423266 5227 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 22:25:10.423802 5658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } }
I20260811 22:25:10.426541 5658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f468027dda432ea0e552fb89c12e61. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.426801 5658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:10.427297 5704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 22:25:10.431205 5704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 22:25:10.447139 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.415494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43334 (local address 127.29.3.126:45707)
0811 22:25:10.415625 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.415630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.417622 (+ 1992us) server_negotiation.cc:407] Connection header received
0811 22:25:10.417828 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.417832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.417881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.417930 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.418995 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.419800 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.421013 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.421355 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.421425 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.436234 (+ 14809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.436238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.436241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.436278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.437957 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.443040 (+ 5083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.443045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.443047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.443149 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.444323 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.444346 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.444348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.444616 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.444753 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.446793 (+ 2040us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.446963 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 22:25:10.448787 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.416278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.29.3.124:45371)
0811 22:25:10.416393 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.416397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.423464 (+ 7067us) server_negotiation.cc:407] Connection header received
0811 22:25:10.423640 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.423643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.423726 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.423779 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.424790 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.425583 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.426757 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.427093 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.427172 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.438561 (+ 11389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.438566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.438568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.438604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.440261 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.443479 (+ 3218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.443483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.443485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.443585 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.445306 (+ 1721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.445311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.445313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.445575 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.445712 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.448462 (+ 2750us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.448605 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 22:25:10.450042 5559 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [term 1 FOLLOWER]: Refusing update from remote peer 10f468027dda432ea0e552fb89c12e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:10.450553 5429 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [term 1 FOLLOWER]: Refusing update from remote peer 10f468027dda432ea0e552fb89c12e61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 22:25:10.450795 5658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:10.451323 5658 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 22:25:10.455449 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f468027dda432ea0e552fb89c12e61. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.455483 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10f468027dda432ea0e552fb89c12e61. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.455606 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.455637 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.456009 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:10.417022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.29.3.125:37199)
0811 22:25:10.417138 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.417142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.428118 (+ 10976us) server_negotiation.cc:407] Connection header received
0811 22:25:10.428309 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.428312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.428371 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.428422 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.429399 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.430172 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.431399 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:10.431770 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:10.431847 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:10.440831 (+ 8984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 22:25:10.440835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 22:25:10.440838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 22:25:10.440872 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 22:25:10.442504 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.443918 (+ 1414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.443923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.443924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.444017 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 22:25:10.446038 (+ 2021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 22:25:10.446042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 22:25:10.446044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 22:25:10.446286 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 22:25:10.446427 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:10.455185 (+ 8758us) server_negotiation.cc:299] Negotiation successful
0811 22:25:10.455380 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 22:25:10.464236 5707 mvcc.cc:205] Tried to move back new op lower bound from 7317451204392960000 to 7317451204302094336. Current Snapshot: MvccSnapshot[applied={T|T < 7317451204392960000}]
I20260811 22:25:10.464790 5658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.464926 5658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: This master's current role is: LEADER
W20260811 22:25:10.465583 5392 tablet.cc:2369] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 22:25:10.466987 5630 sys_catalog.cc:455] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.467120 5630 sys_catalog.cc:458] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.468186 5704 catalog_manager.cc:1347] Generated new cluster ID: d78ef7f16be743fdb9ccbf1a4af133b2
I20260811 22:25:10.468271 5704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 22:25:10.472033 5658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.472173 5658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61 [sys.catalog]: This master's current role is: LEADER
I20260811 22:25:10.473387 5611 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "10f468027dda432ea0e552fb89c12e61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10f468027dda432ea0e552fb89c12e61" member_type: VOTER last_known_addr { host: "127.29.3.126" port: 45707 } } peers { permanent_uuid: "b26a875b11664be58f4a80519334228f" member_type: VOTER last_known_addr { host: "127.29.3.125" port: 37199 } } peers { permanent_uuid: "aab804bcf9e04ee790a4da42d6eb6e75" member_type: VOTER last_known_addr { host: "127.29.3.124" port: 45371 } } }
I20260811 22:25:10.473531 5611 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 22:25:10.473731 5708 mvcc.cc:205] Tried to move back new op lower bound from 7317451204392960000 to 7317451204302094336. Current Snapshot: MvccSnapshot[applied={T|T < 7317451204392960000}]
I20260811 22:25:10.492905 5704 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 22:25:10.494393 5704 catalog_manager.cc:1504] Loading token signing keys...
I20260811 22:25:10.513000 5704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10f468027dda432ea0e552fb89c12e61: Generated new TSK 0
I20260811 22:25:10.514231 5704 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 22:25:10.543871 5352 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.542733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43340 (local address 127.29.3.126:45707)
0811 22:25:10.542857 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.542862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.542879 (+ 17us) server_negotiation.cc:407] Connection header received
0811 22:25:10.542931 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.542934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.542987 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.543062 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.543735 (+ 673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43340: BlockingRecv error: recv got EOF from 127.0.0.1:43340 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260811 22:25:10.543871 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.541900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.29.3.125:37199)
0811 22:25:10.542035 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.542039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.542932 (+ 893us) server_negotiation.cc:407] Connection header received
0811 22:25:10.543102 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.543105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.543161 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.543226 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.543724 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33634: BlockingRecv error: recv got EOF from 127.0.0.1:33634 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 22:25:10.545320 5606 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.542757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.29.3.124:45371)
0811 22:25:10.542912 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.542916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.544398 (+ 1482us) server_negotiation.cc:407] Connection header received
0811 22:25:10.544618 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.544620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.544669 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.544742 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.545161 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60534: BlockingRecv error: recv got EOF from 127.0.0.1:60534 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 22:25:10.548496 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.547183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.29.3.125:37199)
0811 22:25:10.547313 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.547318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.547689 (+ 371us) server_negotiation.cc:407] Connection header received
0811 22:25:10.547833 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.547836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.547903 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.547973 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.548384 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33640: BlockingRecv error: recv got EOF from 127.0.0.1:33640 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 22:25:10.552374 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.551553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.29.3.125:37199)
0811 22:25:10.551667 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.551671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.551683 (+ 12us) server_negotiation.cc:407] Connection header received
0811 22:25:10.551730 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.551731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.551773 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.551839 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.552262 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33650: BlockingRecv error: recv got EOF from 127.0.0.1:33650 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 22:25:10.557444 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.556174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.29.3.125:37199)
0811 22:25:10.556292 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.556296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.556651 (+ 355us) server_negotiation.cc:407] Connection header received
0811 22:25:10.556793 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.556796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.556848 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.556915 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.557323 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33662: BlockingRecv error: recv got EOF from 127.0.0.1:33662 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 22:25:10.564286 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.563034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33670 (local address 127.29.3.125:37199)
0811 22:25:10.563156 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.563161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.563478 (+ 317us) server_negotiation.cc:407] Connection header received
0811 22:25:10.563600 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.563603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.563656 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.563722 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.564151 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33670: BlockingRecv error: recv got EOF from 127.0.0.1:33670 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260811 22:25:10.575750 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.574144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.29.3.125:37199)
0811 22:25:10.574274 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.574280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.574890 (+ 610us) server_negotiation.cc:407] Connection header received
0811 22:25:10.575046 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.575049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.575100 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.575171 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.575623 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33684: BlockingRecv error: recv got EOF from 127.0.0.1:33684 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 22:25:10.595498 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.594127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.29.3.125:37199)
0811 22:25:10.594283 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.594316 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.594606 (+ 290us) server_negotiation.cc:407] Connection header received
0811 22:25:10.594753 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.594756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.594811 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.594881 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.595369 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33690: BlockingRecv error: recv got EOF from 127.0.0.1:33690 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260811 22:25:10.630628 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.629524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.29.3.125:37199)
0811 22:25:10.629674 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.629678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.629767 (+ 89us) server_negotiation.cc:407] Connection header received
0811 22:25:10.629890 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.629893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.629959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.630056 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.630504 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33700: BlockingRecv error: recv got EOF from 127.0.0.1:33700 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 22:25:10.698230 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.697003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.29.3.125:37199)
0811 22:25:10.697158 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.697161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.697263 (+ 102us) server_negotiation.cc:407] Connection header received
0811 22:25:10.697444 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.697447 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.697506 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.697628 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.698117 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33708: BlockingRecv error: recv got EOF from 127.0.0.1:33708 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 22:25:10.829641 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:10.828566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33712 (local address 127.29.3.125:37199)
0811 22:25:10.828761 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:10.828765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:10.828820 (+ 55us) server_negotiation.cc:407] Connection header received
0811 22:25:10.828957 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:10.828960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:10.829011 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:10.829084 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:10.829509 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33712: BlockingRecv error: recv got EOF from 127.0.0.1:33712 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260811 22:25:11.088850 5493 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 22:25:11.087899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33726 (local address 127.29.3.125:37199)
0811 22:25:11.088025 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.088029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.088045 (+ 16us) server_negotiation.cc:407] Connection header received
0811 22:25:11.088153 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.088155 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.088210 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.088283 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 22:25:11.088721 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33726: BlockingRecv error: recv got EOF from 127.0.0.1:33726 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:25:11.296446 5626 catalog_manager.cc:1259] Loaded cluster ID: d78ef7f16be743fdb9ccbf1a4af133b2
I20260811 22:25:11.296546 5626 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: loading cluster ID for follower catalog manager: success
I20260811 22:25:11.298324 5626 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: acquiring CA information for follower catalog manager: success
I20260811 22:25:11.299597 5626 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aab804bcf9e04ee790a4da42d6eb6e75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:25:11.400856 5652 catalog_manager.cc:1259] Loaded cluster ID: d78ef7f16be743fdb9ccbf1a4af133b2
I20260811 22:25:11.400956 5652 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: loading cluster ID for follower catalog manager: success
I20260811 22:25:11.402730 5652 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: acquiring CA information for follower catalog manager: success
I20260811 22:25:11.403976 5652 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b26a875b11664be58f4a80519334228f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 22:25:11.607113 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.603123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.29.3.125:37199)
0811 22:25:11.603377 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.603381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.603494 (+ 113us) server_negotiation.cc:407] Connection header received
0811 22:25:11.603632 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.603635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.603772 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.603874 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:11.604171 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.604720 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.605556 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.605776 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.605845 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.606188 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:11.606704 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:11.606879 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.606913 (+ 34us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.606970 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
I20260811 22:25:11.614238 5851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.610394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.29.3.124:45371)
0811 22:25:11.610678 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.610683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.610698 (+ 15us) server_negotiation.cc:407] Connection header received
0811 22:25:11.610796 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.610798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.610922 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.611032 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:11.611283 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.611838 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.612706 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.612940 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.613032 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.613348 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:11.613824 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:11.613938 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.614007 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.614073 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
I20260811 22:25:11.621610 5845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.617472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33746 (local address 127.29.3.125:37199)
0811 22:25:11.617649 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.617654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.617991 (+ 337us) server_negotiation.cc:407] Connection header received
0811 22:25:11.618113 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.618117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.618180 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.618271 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:11.618535 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.619173 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.620235 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.620595 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.620698 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.621041 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:11.621141 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:11.621221 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.621388 (+ 167us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.621452 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 22:25:11.622215 5851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.617889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.29.3.124:45371)
0811 22:25:11.617996 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.618000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.618388 (+ 388us) server_negotiation.cc:407] Connection header received
0811 22:25:11.618435 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.618437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.618481 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.618543 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:11.618901 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.619659 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.620806 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.621117 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.621204 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.621469 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:11.621561 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:11.621619 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.622027 (+ 408us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.622092 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 22:25:11.622380 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 22:25:11.617472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43346 (local address 127.29.3.126:45707)
0811 22:25:11.617763 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0811 22:25:11.617768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 22:25:11.618062 (+ 294us) server_negotiation.cc:407] Connection header received
0811 22:25:11.618257 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 22:25:11.618261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 22:25:11.618471 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 22:25:11.618598 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 22:25:11.618801 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.619634 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.620670 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 22:25:11.620922 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 22:25:11.621010 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 22:25:11.621419 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 22:25:11.621828 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 22:25:11.621915 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 22:25:11.622030 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 22:25:11.622216 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260811 22:25:11.625705 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5227
I20260811 22:25:11.644845 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5360
I20260811 22:25:11.677881 29709 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskCdqKk8/build/debug/bin/kudu with pid 5495
2026-08-11T22:25:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3474 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6835 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (300744 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST